2022-03-29T17:37:06.636 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-29T17:37:06.638 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2022-03-20_01:01:02-crimson-rados-quincy-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --description crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_python_api_tests_old_format} --archive /home/teuthworker/archive/teuthology-2022-03-20_01:01:02-crimson-rados-quincy-distro-default-smithi/6747525 -- /home/teuthworker/archive/teuthology-2022-03-20_01:01:02-crimson-rados-quincy-distro-default-smithi/6747525/orig.config.yaml 2022-03-29T17:37:06.671 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-29T17:37:06.733 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2022-03-20_01:01:02-crimson-rados-quincy-distro-default-smithi/6747525 branch: quincy description: crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_python_api_tests_old_format} email: ceph-qa@ceph.io first_in_suite: false job_id: '6747525' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2022-03-20_01:01:02-crimson-rados-quincy-distro-default-smithi nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: quincy ceph: conf: global: ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(SLOW_OPS\) - slow request log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: quincy sha1: 60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 owner: scheduled_teuthology@teuthology priority: 1000 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 sleep_before_teardown: 0 suite: crimson-rados suite_branch: quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph_60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 targets: smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+W4RzCfhLa+nXiuw2c9D/5nDX3hHa2L3Nyp6KyrvAaaVIRfQj1I53bs5iq5Q0S2SbftuTF9kTUzWweNmoiWh5/Z5JpipjO7p3lI0NXrX+mf0AzXVfePQk99eVrIqGx8ryoha0QMmm7jo2TPSD18xpR7gogfSMnX8N7IrfncWrOShipp7Jk81PjCxWAtTVz2ghf4zFFtYl88DaV0EC2xZYVkRC9j8vaF0vh7nLjrvtlPKBW1KAgWFGYzlnKM/e2/IUgVSnlg4EpC4pZcJKh1qqR7wctmg6CLYp+LCTldpt9kR4QOdAKSVB2yMbkOoWwLbPtdfBT0O5CyOzc933E/fZEkUPDQPLyKpTOxCqU6XJjCX0aSj3MzDjSk/sW70et1TXFkQBhrGIVmJXWeKGUDFW8wfeUFpg1J6hvxFAgrc35ZvgOgwWo+GrzGXJrk+BBEucHnjCLipGeebztraFCqciOArJNtPJ1p+je3s9z07nBXAH8NK91YOzStkGt2PI1j0= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/test_librbd_python.sh teuthology_branch: master teuthology_sha1: a4e18662e5de936eb0ac089e531b933ecbcadb0a timestamp: 2022-03-20_01:01:02 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-29T17:37:06.734 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/qa; will attempt to use it 2022-03-29T17:37:06.735 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/qa/tasks 2022-03-29T17:37:06.735 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-29T17:37:06.737 INFO:teuthology.task.internal:Checking packages... 2022-03-29T17:37:06.759 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5' 2022-03-29T17:37:06.759 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-29T17:37:06.760 INFO:teuthology.packaging:ref: None 2022-03-29T17:37:06.760 INFO:teuthology.packaging:tag: None 2022-03-29T17:37:06.761 INFO:teuthology.packaging:branch: quincy 2022-03-29T17:37:06.761 INFO:teuthology.packaging:sha1: 60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 2022-03-29T17:37:06.761 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&ref=quincy 2022-03-29T17:37:06.961 INFO:teuthology.task.internal:Found packages for ceph version 17.1.0-123.gc83f2418 2022-03-29T17:37:06.962 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-29T17:37:06.973 INFO:teuthology.task.internal:no buildpackages task found 2022-03-29T17:37:06.974 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-29T17:37:07.027 INFO:teuthology.task.internal:Saving configuration 2022-03-29T17:37:07.038 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-29T17:37:07.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-29T17:37:07.139 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2022-03-20_01:01:02-crimson-rados-quincy-distro-default-smithi/6747525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-29 17:32:22.332747', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXBsCSWdLMeFIVajozYwZxxX14US2xghAVcCwHjiXwbmyS6VMRd/PqnRQJJAqiuP0EnfrXAd3HgaSIUpq60sSs='} 2022-03-29T17:37:07.140 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-29T17:37:07.157 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2022-03-29T17:37:07.157 INFO:teuthology.run_tasks:Running task console_log... 2022-03-29T17:37:07.225 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4a1b6bb8c8>, signals=[15]) 2022-03-29T17:37:07.227 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-29T17:37:07.238 INFO:teuthology.task.internal:Opening connections... 2022-03-29T17:37:07.238 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2022-03-29T17:37:07.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-29T17:37:07.310 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-29T17:37:07.322 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2022-03-29T17:37:07.387 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2022-03-29T17:37:07.387 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2022-03-29T17:37:07.446 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2022-03-29T17:37:07.446 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8" 2022-03-29T17:37:07.447 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2022-03-29T17:37:07.447 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2022-03-29T17:37:07.447 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8" 2022-03-29T17:37:07.447 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8" 2022-03-29T17:37:07.448 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-29T17:37:07.448 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2022-03-29T17:37:07.448 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-29T17:37:07.448 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2022-03-29T17:37:07.449 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-29T17:37:07.449 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-29T17:37:07.449 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-29T17:37:07.450 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2022-03-29T17:37:07.470 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-29T17:37:07.484 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-29T17:37:07.551 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-29T17:37:07.552 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2022-03-29T17:37:07.571 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-29T17:37:07.585 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-29T17:37:07.585 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2022-03-29T17:37:07.641 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-29T17:37:07.676 INFO:teuthology.run_tasks:Running task kernel... 2022-03-29T17:37:07.694 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-29T17:37:07.695 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-03-29T17:37:07.696 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-29T17:37:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-29T17:37:07.715 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2022-03-29T17:37:07.773 INFO:teuthology.orchestra.run.smithi138.stdout:4.18.0-358.el8.x86_64 2022-03-29T17:37:07.774 INFO:teuthology.task.kernel:Running kernel on smithi138: 4.18.0-358.el8.x86_64 2022-03-29T17:37:07.774 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum install -y kernel 2022-03-29T17:37:08.829 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:37:08.830 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:37:10.994 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - AppStream 24 MB/s | 21 MB 00:00 2022-03-29T17:37:15.161 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - Base 39 MB/s | 21 MB 00:00 2022-03-29T17:37:18.259 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - Extras 145 kB/s | 18 kB 00:00 2022-03-29T17:37:18.637 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - PowerTools 17 MB/s | 4.4 MB 00:00 2022-03-29T17:37:19.645 INFO:teuthology.orchestra.run.smithi138.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2022-03-29T17:37:20.713 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2022-03-29T17:37:23.529 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 473 kB/s | 24 kB 00:00 2022-03-29T17:37:26.034 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-4.18.0-358.el8.x86_64 is already installed. 2022-03-29T17:37:26.034 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-03-29T17:37:26.132 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:37:26.133 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:37:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2022-03-29T17:37:26.133 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:37:26.134 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:37:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: kernel x86_64 4.18.0-373.el8 CentOS-Base 8.0 M 2022-03-29T17:37:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core x86_64 4.18.0-373.el8 CentOS-Base 39 M 2022-03-29T17:37:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules x86_64 4.18.0-373.el8 CentOS-Base 32 M 2022-03-29T17:37:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:37:26.135 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:37:26.135 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:37:26.135 INFO:teuthology.orchestra.run.smithi138.stdout:Install 3 Packages 2022-03-29T17:37:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:37:26.136 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 79 M 2022-03-29T17:37:26.136 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 92 M 2022-03-29T17:37:26.136 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:37:26.397 INFO:teuthology.orchestra.run.smithi138.stdout:(1/3): kernel-4.18.0-373.el8.x86_64.rpm 31 MB/s | 8.0 MB 00:00 2022-03-29T17:37:27.305 INFO:teuthology.orchestra.run.smithi138.stdout:(2/3): kernel-modules-4.18.0-373.el8.x86_64.rpm 27 MB/s | 32 MB 00:01 2022-03-29T17:37:27.681 INFO:teuthology.orchestra.run.smithi138.stdout:(3/3): kernel-core-4.18.0-373.el8.x86_64.rpm 25 MB/s | 39 MB 00:01 2022-03-29T17:37:27.684 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:37:27.684 INFO:teuthology.orchestra.run.smithi138.stdout:Total 51 MB/s | 79 MB 00:01 2022-03-29T17:37:28.275 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:37:28.653 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:37:28.653 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:37:30.016 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:37:30.025 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:37:32.606 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:37:33.133 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-core-4.18.0-373.el8.x86_64 1/3 2022-03-29T17:37:36.739 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-4.18.0-373.el8.x86_64 1/3 2022-03-29T17:37:37.151 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-4.18.0-373.el8.x86_64 2/3 2022-03-29T17:37:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-4.18.0-373.el8.x86_64 2/3 2022-03-29T17:37:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-4.18.0-373.el8.x86_64 3/3 2022-03-29T17:38:09.009 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-4.18.0-373.el8.x86_64 3/3 2022-03-29T17:38:09.367 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-4.18.0-373.el8.x86_64 3/3 2022-03-29T17:38:09.368 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-4.18.0-373.el8.x86_64 1/3 2022-03-29T17:38:09.368 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-core-4.18.0-373.el8.x86_64 2/3 2022-03-29T17:38:09.890 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-4.18.0-373.el8.x86_64 3/3 2022-03-29T17:38:09.890 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:38:09.890 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:38:09.890 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-4.18.0-373.el8.x86_64 kernel-core-4.18.0-373.el8.x86_64 2022-03-29T17:38:09.891 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-4.18.0-373.el8.x86_64 2022-03-29T17:38:09.891 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:38:09.891 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:38:09.994 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel --last | head -n 1 2022-03-29T17:38:10.219 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-4.18.0-373.el8.x86_64 Tue 29 Mar 2022 05:37:40 PM UTC 2022-03-29T17:38:10.221 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-373.el8.x86_64 2022-03-29T17:38:10.221 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-358.el8.x86_64 Expected: 4.18.0-373.el8.x86_64 2022-03-29T17:38:10.222 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-29T17:38:10.222 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-29T17:38:10.222 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2022-03-29T17:38:10.223 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2022-03-29T17:38:10.223 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel --last | head -n 1 2022-03-29T17:38:10.395 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-4.18.0-373.el8.x86_64 Tue 29 Mar 2022 05:37:40 PM UTC 2022-03-29T17:38:10.396 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-373.el8.x86_64 2022-03-29T17:38:10.397 DEBUG:teuthology.orchestra.run.smithi138:> sudo rpm -qi grub2-tools 2022-03-29T17:38:10.430 INFO:teuthology.orchestra.run.smithi138.stdout:Name : grub2-tools 2022-03-29T17:38:10.430 INFO:teuthology.orchestra.run.smithi138.stdout:Epoch : 1 2022-03-29T17:38:10.431 INFO:teuthology.orchestra.run.smithi138.stdout:Version : 2.02 2022-03-29T17:38:10.431 INFO:teuthology.orchestra.run.smithi138.stdout:Release : 106.el8 2022-03-29T17:38:10.432 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: x86_64 2022-03-29T17:38:10.432 INFO:teuthology.orchestra.run.smithi138.stdout:Install Date: Fri 04 Feb 2022 04:06:37 PM UTC 2022-03-29T17:38:10.432 INFO:teuthology.orchestra.run.smithi138.stdout:Group : System Environment/Base 2022-03-29T17:38:10.433 INFO:teuthology.orchestra.run.smithi138.stdout:Size : 9265339 2022-03-29T17:38:10.433 INFO:teuthology.orchestra.run.smithi138.stdout:License : GPLv3+ 2022-03-29T17:38:10.434 INFO:teuthology.orchestra.run.smithi138.stdout:Signature : RSA/SHA256, Mon 08 Nov 2021 06:49:52 AM UTC, Key ID 05b555b38483c65d 2022-03-29T17:38:10.434 INFO:teuthology.orchestra.run.smithi138.stdout:Source RPM : grub2-2.02-106.el8.src.rpm 2022-03-29T17:38:10.435 INFO:teuthology.orchestra.run.smithi138.stdout:Build Date : Mon 08 Nov 2021 06:39:57 AM UTC 2022-03-29T17:38:10.435 INFO:teuthology.orchestra.run.smithi138.stdout:Build Host : kbuilder.bsys.centos.org 2022-03-29T17:38:10.436 INFO:teuthology.orchestra.run.smithi138.stdout:Relocations : (not relocatable) 2022-03-29T17:38:10.436 INFO:teuthology.orchestra.run.smithi138.stdout:Packager : CentOS BuildSystem 2022-03-29T17:38:10.436 INFO:teuthology.orchestra.run.smithi138.stdout:Vendor : CentOS 2022-03-29T17:38:10.437 INFO:teuthology.orchestra.run.smithi138.stdout:URL : http://www.gnu.org/software/grub/ 2022-03-29T17:38:10.437 INFO:teuthology.orchestra.run.smithi138.stdout:Summary : Support tools for GRUB. 2022-03-29T17:38:10.438 INFO:teuthology.orchestra.run.smithi138.stdout:Description : 2022-03-29T17:38:10.438 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:38:10.439 INFO:teuthology.orchestra.run.smithi138.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-03-29T17:38:10.439 INFO:teuthology.orchestra.run.smithi138.stdout:customizable bootloader with modular architecture. It supports a rich 2022-03-29T17:38:10.439 INFO:teuthology.orchestra.run.smithi138.stdout:variety of kernel formats, file systems, computer architectures and 2022-03-29T17:38:10.440 INFO:teuthology.orchestra.run.smithi138.stdout:hardware devices. 2022-03-29T17:38:10.440 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:38:10.441 INFO:teuthology.orchestra.run.smithi138.stdout:This subpackage provides tools for support of all platforms. 2022-03-29T17:38:10.443 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-03-29T17:38:10.443 INFO:teuthology.task.kernel:Updating grub on smithi138 to boot 4.18.0-373.el8.x86_64 2022-03-29T17:38:10.444 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-03-29T17:38:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2022-03-29T17:38:13.630 INFO:teuthology.orchestra.run.smithi138.stderr:done 2022-03-29T17:38:13.646 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2022-03-29T17:38:13.673 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.OvF9kluR6E 2022-03-29T17:38:13.674 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OvF9kluR6E 2022-03-29T17:38:13.748 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.OvF9kluR6E 2022-03-29T17:38:13.914 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.OvF9kluR6E is 6KB 2022-03-29T17:38:13.963 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.OvF9kluR6E 2022-03-29T17:38:13.979 DEBUG:teuthology.orchestra.run.smithi138:> sudo /bin/ls /boot/loader/entries || true 2022-03-29T17:38:14.043 INFO:teuthology.orchestra.run.smithi138.stdout:de9f16d36b2643dd85552c363843f45b-0-rescue.conf 2022-03-29T17:38:14.044 INFO:teuthology.orchestra.run.smithi138.stdout:de9f16d36b2643dd85552c363843f45b-4.18.0-305.3.1.el8.x86_64.conf 2022-03-29T17:38:14.044 INFO:teuthology.orchestra.run.smithi138.stdout:de9f16d36b2643dd85552c363843f45b-4.18.0-358.el8.x86_64.conf 2022-03-29T17:38:14.044 INFO:teuthology.orchestra.run.smithi138.stdout:de9f16d36b2643dd85552c363843f45b-4.18.0-373.el8.x86_64.conf 2022-03-29T17:38:14.045 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-set-default de9f16d36b2643dd85552c363843f45b-4.18.0-373.el8.x86_64 2022-03-29T17:38:14.219 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2022-03-29T17:38:44.250 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2022-03-29T17:38:44.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-03-29T17:38:44.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-29T17:39:02.731 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2022-03-29T17:39:32.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-03-29T17:39:32.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-29T17:39:48.972 DEBUG:teuthology.orchestra.run.smithi138:> true 2022-03-29T17:39:49.262 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-03-29T17:39:49.262 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-373.el8.x86_64"... 2022-03-29T17:39:49.262 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2022-03-29T17:39:49.278 INFO:teuthology.orchestra.run.smithi138.stdout:4.18.0-373.el8.x86_64 2022-03-29T17:39:49.279 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-373.el8.x86_64 vs 4.18.0-373.el8.x86_64 2022-03-29T17:39:49.279 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-29T17:39:49.279 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2022-03-29T17:39:50.280 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-03-29T17:39:50.281 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-29T17:39:50.362 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2022-03-29T17:39:50.389 DEBUG:teuthology.parallel:result is None 2022-03-29T17:39:50.389 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-29T17:39:50.401 INFO:teuthology.task.internal:Creating test directory... 2022-03-29T17:39:50.402 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-29T17:39:50.420 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-29T17:39:50.444 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-29T17:39:50.478 INFO:teuthology.task.internal:Creating archive directory... 2022-03-29T17:39:50.478 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-29T17:39:50.509 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-29T17:39:50.520 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-29T17:39:50.521 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-29T17:39:50.581 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-29T17:39:50.592 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-29T17:39:50.594 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-29T17:39:50.605 INFO:teuthology.task.internal:Configuring sudo... 2022-03-29T17:39:50.606 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-29T17:39:50.662 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-29T17:39:50.675 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-29T17:39:50.676 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-29T17:39:50.719 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-29T17:39:50.806 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-29T17:39:50.857 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-29T17:39:50.939 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-29T17:39:50.983 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:39:50.983 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-29T17:39:51.081 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2022-03-29T17:39:51.197 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-03-29T17:39:53.962 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-29T17:39:53.977 INFO:teuthology.task.internal:Starting timer... 2022-03-29T17:39:53.978 INFO:teuthology.run_tasks:Running task pcp... 2022-03-29T17:39:54.029 INFO:teuthology.run_tasks:Running task selinux... 2022-03-29T17:39:54.061 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-03-29T17:39:54.062 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2022-03-29T17:39:54.120 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2022-03-29T17:39:54.122 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-29T17:39:54.122 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2022-03-29T17:39:54.145 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2022-03-29T17:39:54.146 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi138.front.sepia.ceph.com': 'permissive'} 2022-03-29T17:39:54.147 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-03-29T17:39:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-29T17:39:54.211 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-29T17:39:54.212 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2022-03-29T17:39:54.281 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-29T17:39:54.297 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-29T17:39:54.298 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-29T17:39:54.317 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-29T17:39:54.319 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-29T17:44:51.510 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2022-03-29T17:44:51.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-03-29T17:44:51.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-29T17:44:51.587 DEBUG:teuthology.orchestra.run.smithi138:> true 2022-03-29T17:44:51.657 INFO:teuthology.run_tasks:Running task clock... 2022-03-29T17:44:51.679 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-29T17:44:51.680 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-29T17:44:51.683 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-29T17:44:51.729 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-03-29T17:44:51.744 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-03-29T17:44:51.825 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2022-03-29T17:44:51.836 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2022-03-29T17:44:51.850 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-03-29T17:44:51.864 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-03-29T17:44:51.932 INFO:teuthology.orchestra.run.smithi138.stderr:bash: ntpq: command not found 2022-03-29T17:44:51.938 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-03-29T17:44:51.938 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2022-03-29T17:44:51.939 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-29T17:44:51.939 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-29T17:44:51.940 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-29T17:44:51.940 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-29T17:44:51.940 INFO:teuthology.run_tasks:Running task install... 2022-03-29T17:44:51.953 DEBUG:teuthology.task.install:project ceph 2022-03-29T17:44:51.954 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5'}} 2022-03-29T17:44:51.955 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5'} 2022-03-29T17:44:51.955 INFO:teuthology.task.install:Using flavor: crimson 2022-03-29T17:44:51.966 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-29T17:44:51.967 INFO:teuthology.task.install:extra packages: [] 2022-03-29T17:44:51.968 DEBUG:teuthology.orchestra.run.smithi138:> sudo lsb_release -is 2022-03-29T17:44:52.013 INFO:teuthology.orchestra.run.smithi138.stdout:CentOSStream 2022-03-29T17:44:52.014 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-03-29T17:44:52.015 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5', 'tag': None, 'wait_for_package': False} 2022-03-29T17:44:52.016 DEBUG:teuthology.orchestra.run.smithi138:> sudo lsb_release -is 2022-03-29T17:44:52.089 INFO:teuthology.orchestra.run.smithi138.stdout:CentOSStream 2022-03-29T17:44:52.090 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-03-29T17:44:52.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 2022-03-29T17:44:52.245 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/centos/8/flavors/crimson/ 2022-03-29T17:44:52.245 INFO:teuthology.task.install.rpm:Package version is 17.1.0-56.g60fdd357 2022-03-29T17:44:52.382 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/centos/8/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/centos/8/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/centos/8/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-03-29T17:44:52.383 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:44:52.383 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-03-29T17:44:52.410 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-03-29T17:44:52.411 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-03-29T17:44:52.478 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-03-29T17:44:52.556 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-03-29T17:44:52.623 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2022-03-29T17:44:52.625 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2022-03-29T17:44:52.841 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:44:52.841 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:44:52.856 INFO:teuthology.orchestra.run.smithi138.stdout:65 files removed 2022-03-29T17:44:52.876 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw 2022-03-29T17:44:53.052 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:44:53.053 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:44:53.508 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 287 kB/s | 80 kB 00:00 2022-03-29T17:44:53.800 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 35 kB/s | 9.1 kB 00:00 2022-03-29T17:44:54.076 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 6.1 kB/s | 1.5 kB 00:00 2022-03-29T17:44:54.712 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - AppStream 34 MB/s | 21 MB 00:00 2022-03-29T17:44:58.896 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - Base 41 MB/s | 21 MB 00:00 2022-03-29T17:45:01.995 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - Extras 164 kB/s | 18 kB 00:00 2022-03-29T17:45:02.348 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - PowerTools 19 MB/s | 4.4 MB 00:00 2022-03-29T17:45:03.439 INFO:teuthology.orchestra.run.smithi138.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2022-03-29T17:45:05.032 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 11 MB 00:01 2022-03-29T17:45:07.715 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 440 kB/s | 24 kB 00:00 2022-03-29T17:45:10.375 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:45:10.377 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:45:10.377 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:45:10.378 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:45:10.378 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:45:10.378 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:17.1.0-56.g60fdd357.el8 ceph 9.7 M 2022-03-29T17:45:10.378 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-03-29T17:45:10.379 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:17.1.0-56.g60fdd357.el8 ceph 6.3 M 2022-03-29T17:45:10.379 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:17.1.0-56.g60fdd357.el8 ceph 22 M 2022-03-29T17:45:10.379 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:17.1.0-56.g60fdd357.el8 ceph 24 k 2022-03-29T17:45:10.379 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-03-29T17:45:10.380 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:17.1.0-56.g60fdd357.el8 ceph 785 k 2022-03-29T17:45:10.380 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-03-29T17:45:10.380 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2022-03-29T17:45:10.380 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:17.1.0-56.g60fdd357.el8 ceph 3.4 M 2022-03-29T17:45:10.380 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:17.1.0-56.g60fdd357.el8 ceph 493 k 2022-03-29T17:45:10.381 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:17.1.0-56.g60fdd357.el8 ceph 3.7 M 2022-03-29T17:45:10.381 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-03-29T17:45:10.381 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:17.1.0-56.g60fdd357.el8 ceph 3.9 M 2022-03-29T17:45:10.381 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-03-29T17:45:10.381 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-03-29T17:45:10.382 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse x86_64 2:17.1.0-56.g60fdd357.el8 ceph 45 k 2022-03-29T17:45:10.382 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:17.1.0-56.g60fdd357.el8 ceph 104 k 2022-03-29T17:45:10.382 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:17.1.0-56.g60fdd357.el8 ceph 197 k 2022-03-29T17:45:10.382 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-03-29T17:45:10.383 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:17.1.0-56.g60fdd357.el8 ceph 409 k 2022-03-29T17:45:10.383 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:17.1.0-56.g60fdd357.el8 ceph 391 k 2022-03-29T17:45:10.383 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:17.1.0-56.g60fdd357.el8 ceph 119 k 2022-03-29T17:45:10.383 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:45:10.383 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:45:10.384 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:45:10.384 INFO:teuthology.orchestra.run.smithi138.stdout:Install 22 Packages 2022-03-29T17:45:10.384 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:45:10.385 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 52 M 2022-03-29T17:45:10.385 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 264 M 2022-03-29T17:45:10.385 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:45:11.192 INFO:teuthology.orchestra.run.smithi138.stdout:(1/22): ceph-base-17.1.0-56.g60fdd357.el8.x86_6 7.8 MB/s | 6.3 MB 00:00 2022-03-29T17:45:11.301 INFO:teuthology.orchestra.run.smithi138.stdout:(2/22): ceph-selinux-17.1.0-56.g60fdd357.el8.x8 223 kB/s | 24 kB 00:00 2022-03-29T17:45:11.743 INFO:teuthology.orchestra.run.smithi138.stdout:(3/22): ceph-radosgw-17.1.0-56.g60fdd357.el8.x8 7.1 MB/s | 9.7 MB 00:01 2022-03-29T17:45:11.768 INFO:teuthology.orchestra.run.smithi138.stdout:(4/22): libcephfs2-17.1.0-56.g60fdd357.el8.x86_ 1.6 MB/s | 785 kB 00:00 2022-03-29T17:45:11.893 INFO:teuthology.orchestra.run.smithi138.stdout:(5/22): libradosstriper1-17.1.0-56.g60fdd357.el 3.9 MB/s | 493 kB 00:00 2022-03-29T17:45:12.336 INFO:teuthology.orchestra.run.smithi138.stdout:(6/22): ceph-common-17.1.0-56.g60fdd357.el8.x86 11 MB/s | 22 MB 00:01 2022-03-29T17:45:12.403 INFO:teuthology.orchestra.run.smithi138.stdout:(7/22): librados2-17.1.0-56.g60fdd357.el8.x86_6 5.1 MB/s | 3.4 MB 00:00 2022-03-29T17:45:12.487 INFO:teuthology.orchestra.run.smithi138.stdout:(8/22): librbd1-17.1.0-56.g60fdd357.el8.x86_64. 6.2 MB/s | 3.7 MB 00:00 2022-03-29T17:45:12.503 INFO:teuthology.orchestra.run.smithi138.stdout:(9/22): python3-ceph-argparse-17.1.0-56.g60fdd3 445 kB/s | 45 kB 00:00 2022-03-29T17:45:12.570 INFO:teuthology.orchestra.run.smithi138.stdout:(10/22): python3-ceph-common-17.1.0-56.g60fdd35 1.2 MB/s | 104 kB 00:00 2022-03-29T17:45:12.595 INFO:teuthology.orchestra.run.smithi138.stdout:(11/22): python3-cephfs-17.1.0-56.g60fdd357.el8 2.1 MB/s | 197 kB 00:00 2022-03-29T17:45:12.671 INFO:teuthology.orchestra.run.smithi138.stdout:(12/22): python3-rados-17.1.0-56.g60fdd357.el8. 4.0 MB/s | 409 kB 00:00 2022-03-29T17:45:12.738 INFO:teuthology.orchestra.run.smithi138.stdout:(13/22): python3-rgw-17.1.0-56.g60fdd357.el8.x8 1.8 MB/s | 119 kB 00:00 2022-03-29T17:45:12.797 INFO:teuthology.orchestra.run.smithi138.stdout:(14/22): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2022-03-29T17:45:12.863 INFO:teuthology.orchestra.run.smithi138.stdout:(15/22): librgw2-17.1.0-56.g60fdd357.el8.x86_64 7.4 MB/s | 3.9 MB 00:00 2022-03-29T17:45:12.897 INFO:teuthology.orchestra.run.smithi138.stdout:(16/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2022-03-29T17:45:12.922 INFO:teuthology.orchestra.run.smithi138.stdout:(17/22): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.9 MB/s | 47 kB 00:00 2022-03-29T17:45:12.949 INFO:teuthology.orchestra.run.smithi138.stdout:(18/22): python3-prettytable-0.7.2-14.el8.noarc 523 kB/s | 44 kB 00:00 2022-03-29T17:45:13.082 INFO:teuthology.orchestra.run.smithi138.stdout:(19/22): liboath-2.6.2-3.el8.x86_64.rpm 440 kB/s | 59 kB 00:00 2022-03-29T17:45:13.124 INFO:teuthology.orchestra.run.smithi138.stdout:(20/22): python3-rbd-17.1.0-56.g60fdd357.el8.x8 740 kB/s | 391 kB 00:00 2022-03-29T17:45:13.149 INFO:teuthology.orchestra.run.smithi138.stdout:(21/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2022-03-29T17:45:13.175 INFO:teuthology.orchestra.run.smithi138.stdout:(22/22): libunwind-1.3.1-3.el8.x86_64.rpm 814 kB/s | 75 kB 00:00 2022-03-29T17:45:13.178 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:45:13.178 INFO:teuthology.orchestra.run.smithi138.stdout:Total 19 MB/s | 52 MB 00:02 2022-03-29T17:45:13.219 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:45:13.262 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:45:13.263 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:45:13.567 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:45:13.570 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:45:14.552 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:45:14.742 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/22 2022-03-29T17:45:14.938 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/22 2022-03-29T17:45:15.343 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/22 2022-03-29T17:45:15.505 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados2-2:17.1.0-56.g60fdd357.el8.x86_64 3/22 2022-03-29T17:45:15.593 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:17.1.0-56.g60fdd357.el8.x86_64 3/22 2022-03-29T17:45:15.831 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:17.1.0-56.g60fdd357.el8.x86_64 4/22 2022-03-29T17:45:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:17.1.0-56.g60fdd357.el8.x86_64 5/22 2022-03-29T17:45:16.449 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:17.1.0-56.g60fdd357.el8.x86_64 5/22 2022-03-29T17:45:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librbd1-2:17.1.0-56.g60fdd357.el8.x86_64 6/22 2022-03-29T17:45:16.721 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:17.1.0-56.g60fdd357.el8.x86_64 6/22 2022-03-29T17:45:16.913 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/22 2022-03-29T17:45:17.114 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/22 2022-03-29T17:45:17.531 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/22 2022-03-29T17:45:17.738 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:17.1.0-56.g60fdd357.el8.x86_64 9/22 2022-03-29T17:45:17.802 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:17.1.0-56.g60fdd357.el8.x86_64 9/22 2022-03-29T17:45:17.942 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:17.1.0-56.g60fdd357.el8.x8 10/22 2022-03-29T17:45:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:17.1.0-56.g60fdd357.el8.x86_64 11/22 2022-03-29T17:45:18.358 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:17.1.0-56.g60fdd357.el8.x86_64 12/22 2022-03-29T17:45:18.584 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:17.1.0-56.g60fdd357.el8.x86_64 13/22 2022-03-29T17:45:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:17.1.0-56.g60fdd357.el8.x86_64 14/22 2022-03-29T17:45:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:17.1.0-56.g60fdd357.el8.x86_64 14/22 2022-03-29T17:45:19.014 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/22 2022-03-29T17:45:19.204 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/22 2022-03-29T17:45:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/22 2022-03-29T17:45:19.605 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:17.1.0-56.g60fdd357.el8.x86_ 18/22 2022-03-29T17:45:23.207 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:17.1.0-56.g60fdd357.el8.x86_64 19/22 2022-03-29T17:45:23.463 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:17.1.0-56.g60fdd357.el8.x86_64 19/22 2022-03-29T17:45:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:17.1.0-56.g60fdd357.el8.x86_64 19/22 2022-03-29T17:45:24.460 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:17.1.0-56.g60fdd357.el8.x86_64 20/22 2022-03-29T17:45:24.555 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:17.1.0-56.g60fdd357.el8.x86_64 20/22 2022-03-29T17:45:24.704 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:17.1.0-56.g60fdd357.el8.x86_64 21/22 2022-03-29T17:45:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:17.1.0-56.g60fdd357.el8.x86_64 21/22 2022-03-29T17:45:43.606 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2022-03-29T17:45:43.606 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2022-03-29T17:45:43.606 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /dev 2022-03-29T17:45:43.607 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /run 2022-03-29T17:45:43.607 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2022-03-29T17:45:43.607 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2022-03-29T17:45:43.607 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2022-03-29T17:45:43.608 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2022-03-29T17:45:43.608 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /dev 2022-03-29T17:45:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:45:44.807 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:17.1.0-56.g60fdd357.el8.x86_64 22/22 2022-03-29T17:45:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:17.1.0-56.g60fdd357.el8.x86_64 22/22 2022-03-29T17:45:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:17.1.0-56.g60fdd357.el8.x86_64 1/22 2022-03-29T17:45:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:17.1.0-56.g60fdd357.el8.x86_64 2/22 2022-03-29T17:45:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:17.1.0-56.g60fdd357.el8.x86_64 3/22 2022-03-29T17:45:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:17.1.0-56.g60fdd357.el8.x86_64 4/22 2022-03-29T17:45:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:17.1.0-56.g60fdd357.el8.x86_64 5/22 2022-03-29T17:45:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:17.1.0-56.g60fdd357.el8.x86_64 6/22 2022-03-29T17:45:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:17.1.0-56.g60fdd357.el8.x86_64 7/22 2022-03-29T17:45:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:17.1.0-56.g60fdd357.el8.x86_64 8/22 2022-03-29T17:45:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:17.1.0-56.g60fdd357.el8.x86_64 9/22 2022-03-29T17:45:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:17.1.0-56.g60fdd357.el8.x8 10/22 2022-03-29T17:45:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:17.1.0-56.g60fdd357.el8.x86_ 11/22 2022-03-29T17:45:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:17.1.0-56.g60fdd357.el8.x86_64 12/22 2022-03-29T17:45:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:17.1.0-56.g60fdd357.el8.x86_64 13/22 2022-03-29T17:45:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:17.1.0-56.g60fdd357.el8.x86_64 14/22 2022-03-29T17:45:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:17.1.0-56.g60fdd357.el8.x86_64 15/22 2022-03-29T17:45:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 16/22 2022-03-29T17:45:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 17/22 2022-03-29T17:45:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/22 2022-03-29T17:45:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/22 2022-03-29T17:45:45.301 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 20/22 2022-03-29T17:45:45.301 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/22 2022-03-29T17:45:46.229 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/22 2022-03-29T17:45:46.229 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:45:46.230 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:45:46.230 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.230 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.230 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.231 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.231 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-03-29T17:45:46.231 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.231 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.2-3.el8.x86_64 2022-03-29T17:45:46.231 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-03-29T17:45:46.232 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.232 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.232 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.232 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-03-29T17:45:46.233 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.233 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.3.1-3.el8.x86_64 2022-03-29T17:45:46.233 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-03-29T17:45:46.233 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.234 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.234 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.234 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-03-29T17:45:46.234 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.235 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.235 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:45:46.235 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:45:46.235 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:45:46.473 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-test 2022-03-29T17:45:46.669 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:45:46.669 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:45:47.069 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:40 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:45:47.936 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:45:47.938 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:45:47.938 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:45:47.938 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:45:47.939 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:45:47.939 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:17.1.0-56.g60fdd357.el8 ceph 146 M 2022-03-29T17:45:47.939 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-03-29T17:45:47.939 INFO:teuthology.orchestra.run.smithi138.stdout: cryptopp x86_64 8.6.0-1.el8 epel 1.3 M 2022-03-29T17:45:47.940 INFO:teuthology.orchestra.run.smithi138.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-03-29T17:45:47.940 INFO:teuthology.orchestra.run.smithi138.stdout: libasan x86_64 8.5.0-10.el8 CentOS-Base 412 k 2022-03-29T17:45:47.940 INFO:teuthology.orchestra.run.smithi138.stdout: libatomic x86_64 8.5.0-10.el8 CentOS-Base 24 k 2022-03-29T17:45:47.940 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:17.1.0-56.g60fdd357.el8 ceph 171 k 2022-03-29T17:45:47.940 INFO:teuthology.orchestra.run.smithi138.stdout: libubsan x86_64 8.5.0-10.el8 CentOS-Base 151 k 2022-03-29T17:45:47.941 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-03-29T17:45:47.941 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-03-29T17:45:47.941 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-03-29T17:45:47.941 INFO:teuthology.orchestra.run.smithi138.stdout: yaml-cpp x86_64 0.6.2-3.el8 epel 171 k 2022-03-29T17:45:47.942 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:45:47.942 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:45:47.942 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:45:47.942 INFO:teuthology.orchestra.run.smithi138.stdout:Install 11 Packages 2022-03-29T17:45:47.942 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:45:47.943 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 149 M 2022-03-29T17:45:47.944 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 1.7 G 2022-03-29T17:45:47.944 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:45:48.006 INFO:teuthology.orchestra.run.smithi138.stdout:(1/11): jq-1.6-3.el8.x86_64.rpm 3.1 MB/s | 202 kB 00:00 2022-03-29T17:45:48.031 INFO:teuthology.orchestra.run.smithi138.stdout:(2/11): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2022-03-29T17:45:48.073 INFO:teuthology.orchestra.run.smithi138.stdout:(3/11): socat-1.7.4.1-1.el8.x86_64.rpm 9.4 MB/s | 323 kB 00:00 2022-03-29T17:45:48.116 INFO:teuthology.orchestra.run.smithi138.stdout:(4/11): libasan-8.5.0-10.el8.x86_64.rpm 9.6 MB/s | 412 kB 00:00 2022-03-29T17:45:48.141 INFO:teuthology.orchestra.run.smithi138.stdout:(5/11): libatomic-8.5.0-10.el8.x86_64.rpm 1.0 MB/s | 24 kB 00:00 2022-03-29T17:45:48.166 INFO:teuthology.orchestra.run.smithi138.stdout:(6/11): libubsan-8.5.0-10.el8.x86_64.rpm 5.9 MB/s | 151 kB 00:00 2022-03-29T17:45:48.468 INFO:teuthology.orchestra.run.smithi138.stdout:(7/11): cryptopp-8.6.0-1.el8.x86_64.rpm 4.2 MB/s | 1.3 MB 00:00 2022-03-29T17:45:48.525 INFO:teuthology.orchestra.run.smithi138.stdout:(8/11): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2022-03-29T17:45:48.559 INFO:teuthology.orchestra.run.smithi138.stdout:(9/11): libcephsqlite-17.1.0-56.g60fdd357.el8.x 277 kB/s | 171 kB 00:00 2022-03-29T17:45:48.584 INFO:teuthology.orchestra.run.smithi138.stdout:(10/11): yaml-cpp-0.6.2-3.el8.x86_64.rpm 2.9 MB/s | 171 kB 00:00 2022-03-29T17:46:19.640 INFO:teuthology.orchestra.run.smithi138.stdout:(11/11): ceph-test-17.1.0-56.g60fdd357.el8.x86_ 4.6 MB/s | 146 MB 00:31 2022-03-29T17:46:19.642 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:46:19.643 INFO:teuthology.orchestra.run.smithi138.stdout:Total 4.7 MB/s | 149 MB 00:31 2022-03-29T17:46:19.704 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:46:19.727 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:46:19.727 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:46:20.394 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:46:20.397 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:46:21.380 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:46:21.580 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : yaml-cpp-0.6.2-3.el8.x86_64 1/11 2022-03-29T17:46:21.860 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/11 2022-03-29T17:46:22.106 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cryptopp-8.6.0-1.el8.x86_64 3/11 2022-03-29T17:46:22.236 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libubsan-8.5.0-10.el8.x86_64 4/11 2022-03-29T17:46:22.304 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libubsan-8.5.0-10.el8.x86_64 4/11 2022-03-29T17:46:22.420 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libatomic-8.5.0-10.el8.x86_64 5/11 2022-03-29T17:46:22.510 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libatomic-8.5.0-10.el8.x86_64 5/11 2022-03-29T17:46:22.686 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libasan-8.5.0-10.el8.x86_64 6/11 2022-03-29T17:46:22.766 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libasan-8.5.0-10.el8.x86_64 6/11 2022-03-29T17:46:22.972 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 7/11 2022-03-29T17:46:23.161 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 8/11 2022-03-29T17:46:23.223 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 8/11 2022-03-29T17:46:23.402 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : jq-1.6-3.el8.x86_64 9/11 2022-03-29T17:46:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:17.1.0-56.g60fdd357.el8.x86_64 10/11 2022-03-29T17:46:46.704 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:17.1.0-56.g60fdd357.el8.x86_64 10/11 2022-03-29T17:46:47.006 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:17.1.0-56.g60fdd357.el8.x86_64 11/11 2022-03-29T17:46:47.274 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-test-2:17.1.0-56.g60fdd357.el8.x86_64 11/11 2022-03-29T17:46:47.274 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:17.1.0-56.g60fdd357.el8.x86_64 1/11 2022-03-29T17:46:47.274 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:17.1.0-56.g60fdd357.el8.x86_64 2/11 2022-03-29T17:46:47.275 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : jq-1.6-3.el8.x86_64 3/11 2022-03-29T17:46:47.275 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/11 2022-03-29T17:46:47.275 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/11 2022-03-29T17:46:47.275 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libasan-8.5.0-10.el8.x86_64 6/11 2022-03-29T17:46:47.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libatomic-8.5.0-10.el8.x86_64 7/11 2022-03-29T17:46:47.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libubsan-8.5.0-10.el8.x86_64 8/11 2022-03-29T17:46:47.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cryptopp-8.6.0-1.el8.x86_64 9/11 2022-03-29T17:46:47.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 10/11 2022-03-29T17:46:47.750 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : yaml-cpp-0.6.2-3.el8.x86_64 11/11 2022-03-29T17:46:47.750 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:46:47.751 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:46:47.751 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:46:47.751 INFO:teuthology.orchestra.run.smithi138.stdout: cryptopp-8.6.0-1.el8.x86_64 2022-03-29T17:46:47.751 INFO:teuthology.orchestra.run.smithi138.stdout: jq-1.6-3.el8.x86_64 2022-03-29T17:46:47.752 INFO:teuthology.orchestra.run.smithi138.stdout: libasan-8.5.0-10.el8.x86_64 2022-03-29T17:46:47.752 INFO:teuthology.orchestra.run.smithi138.stdout: libatomic-8.5.0-10.el8.x86_64 2022-03-29T17:46:47.752 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:46:47.752 INFO:teuthology.orchestra.run.smithi138.stdout: libubsan-8.5.0-10.el8.x86_64 2022-03-29T17:46:47.752 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-03-29T17:46:47.753 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-1.el8.x86_64 2022-03-29T17:46:47.753 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-03-29T17:46:47.754 INFO:teuthology.orchestra.run.smithi138.stdout: yaml-cpp-0.6.2-3.el8.x86_64 2022-03-29T17:46:47.754 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:46:47.754 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:46:47.936 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph 2022-03-29T17:46:48.117 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:46:48.117 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:46:48.530 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:41 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:46:49.357 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:46:49.361 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:46:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:46:49.361 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:46:49.361 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:46:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:17.1.0-56.g60fdd357.el8 ceph 6.4 k 2022-03-29T17:46:49.362 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-03-29T17:46:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:17.1.0-56.g60fdd357.el8 ceph 2.3 M 2022-03-29T17:46:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:17.1.0-56.g60fdd357.el8 ceph 1.5 M 2022-03-29T17:46:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 224 k 2022-03-29T17:46:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:17.1.0-56.g60fdd357.el8 ceph 7.4 M 2022-03-29T17:46:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:17.1.0-56.g60fdd357.el8 ceph 154 M 2022-03-29T17:46:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-03-29T17:46:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-03-29T17:46:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2022-03-29T17:46:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-03-29T17:46:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-03-29T17:46:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2022-03-29T17:46:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-03-29T17:46:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools 2022-03-29T17:46:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: noarch 2.0-4.el8 epel 18 k 2022-03-29T17:46:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-03-29T17:46:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-03-29T17:46:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-03-29T17:46:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-03-29T17:46:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-03-29T17:46:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-03-29T17:46:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-03-29T17:46:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2022-03-29T17:46:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-03-29T17:46:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-03-29T17:46:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-03-29T17:46:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-03-29T17:46:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-03-29T17:46:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-03-29T17:46:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-03-29T17:46:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-03-29T17:46:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-03-29T17:46:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-03-29T17:46:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:46:49.370 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:46:49.370 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:46:49.370 INFO:teuthology.orchestra.run.smithi138.stdout:Install 32 Packages 2022-03-29T17:46:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:46:49.371 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 169 M 2022-03-29T17:46:49.372 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 2.3 G 2022-03-29T17:46:49.372 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:46:49.680 INFO:teuthology.orchestra.run.smithi138.stdout:(1/32): ceph-17.1.0-56.g60fdd357.el8.x86_64.rpm 21 kB/s | 6.4 kB 00:00 2022-03-29T17:46:50.555 INFO:teuthology.orchestra.run.smithi138.stdout:(2/32): ceph-mgr-17.1.0-56.g60fdd357.el8.x86_64 1.3 MB/s | 1.5 MB 00:01 2022-03-29T17:46:50.864 INFO:teuthology.orchestra.run.smithi138.stdout:(3/32): ceph-mds-17.1.0-56.g60fdd357.el8.x86_64 1.5 MB/s | 2.3 MB 00:01 2022-03-29T17:46:50.956 INFO:teuthology.orchestra.run.smithi138.stdout:(4/32): ceph-mgr-modules-core-17.1.0-56.g60fdd3 2.4 MB/s | 224 kB 00:00 2022-03-29T17:46:51.014 INFO:teuthology.orchestra.run.smithi138.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-03-29T17:46:51.040 INFO:teuthology.orchestra.run.smithi138.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-03-29T17:46:51.065 INFO:teuthology.orchestra.run.smithi138.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2022-03-29T17:46:51.091 INFO:teuthology.orchestra.run.smithi138.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-03-29T17:46:51.133 INFO:teuthology.orchestra.run.smithi138.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 11 MB/s | 457 kB 00:00 2022-03-29T17:46:51.158 INFO:teuthology.orchestra.run.smithi138.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.4 MB/s | 237 kB 00:00 2022-03-29T17:46:51.192 INFO:teuthology.orchestra.run.smithi138.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2022-03-29T17:46:51.217 INFO:teuthology.orchestra.run.smithi138.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2022-03-29T17:46:51.426 INFO:teuthology.orchestra.run.smithi138.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 209 kB/s | 44 kB 00:00 2022-03-29T17:46:51.609 INFO:teuthology.orchestra.run.smithi138.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2022-03-29T17:46:51.759 INFO:teuthology.orchestra.run.smithi138.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2022-03-29T17:46:52.018 INFO:teuthology.orchestra.run.smithi138.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2022-03-29T17:46:52.077 INFO:teuthology.orchestra.run.smithi138.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2022-03-29T17:46:52.134 INFO:teuthology.orchestra.run.smithi138.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 312 kB/s | 18 kB 00:00 2022-03-29T17:46:52.176 INFO:teuthology.orchestra.run.smithi138.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 1.1 MB/s | 49 kB 00:00 2022-03-29T17:46:52.226 INFO:teuthology.orchestra.run.smithi138.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.2 MB/s | 59 kB 00:00 2022-03-29T17:46:52.393 INFO:teuthology.orchestra.run.smithi138.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2022-03-29T17:46:52.437 INFO:teuthology.orchestra.run.smithi138.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 374 kB/s | 16 kB 00:00 2022-03-29T17:46:52.485 INFO:teuthology.orchestra.run.smithi138.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 376 kB/s | 19 kB 00:00 2022-03-29T17:46:52.552 INFO:teuthology.orchestra.run.smithi138.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 358 kB/s | 24 kB 00:00 2022-03-29T17:46:52.594 INFO:teuthology.orchestra.run.smithi138.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 705 kB/s | 29 kB 00:00 2022-03-29T17:46:52.636 INFO:teuthology.orchestra.run.smithi138.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 661 kB/s | 27 kB 00:00 2022-03-29T17:46:52.745 INFO:teuthology.orchestra.run.smithi138.stdout:(27/32): python3-waitress-1.2.1-2.el8.1.noarch. 2.2 MB/s | 241 kB 00:00 2022-03-29T17:46:52.862 INFO:teuthology.orchestra.run.smithi138.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2022-03-29T17:46:52.912 INFO:teuthology.orchestra.run.smithi138.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2022-03-29T17:46:52.962 INFO:teuthology.orchestra.run.smithi138.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 459 kB/s | 23 kB 00:00 2022-03-29T17:46:54.488 INFO:teuthology.orchestra.run.smithi138.stdout:(31/32): ceph-mon-17.1.0-56.g60fdd357.el8.x86_6 1.5 MB/s | 7.4 MB 00:04 2022-03-29T17:47:23.686 INFO:teuthology.orchestra.run.smithi138.stdout:(32/32): ceph-osd-17.1.0-56.g60fdd357.el8.x86_6 4.7 MB/s | 154 MB 00:33 2022-03-29T17:47:23.688 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:47:23.689 INFO:teuthology.orchestra.run.smithi138.stdout:Total 4.9 MB/s | 169 MB 00:34 2022-03-29T17:47:23.758 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:47:23.795 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:47:23.795 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:47:24.590 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:47:24.593 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:47:26.324 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:47:26.548 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-03-29T17:47:26.776 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-03-29T17:47:27.021 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-03-29T17:47:27.241 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-03-29T17:47:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-03-29T17:47:27.618 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-03-29T17:47:27.812 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-03-29T17:47:27.965 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-03-29T17:47:28.231 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-03-29T17:47:28.443 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-03-29T17:47:28.741 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-03-29T17:47:28.982 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-03-29T17:47:29.361 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-03-29T17:47:29.595 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-03-29T17:47:29.817 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-03-29T17:47:30.050 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-03-29T17:47:30.285 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-03-29T17:47:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-03-29T17:47:30.847 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-03-29T17:47:31.086 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-03-29T17:47:31.264 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-03-29T17:47:31.559 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-03-29T17:47:31.804 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-03-29T17:47:32.054 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-03-29T17:47:32.392 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-03-29T17:47:32.745 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-03-29T17:47:33.116 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:17.1.0-56.g60fdd357.el8.no 27/32 2022-03-29T17:47:33.306 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:17.1.0-56.g60fdd357.el8.x86_64 28/32 2022-03-29T17:48:01.596 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:17.1.0-56.g60fdd357.el8.x86_64 28/32 2022-03-29T17:48:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:17.1.0-56.g60fdd357.el8.x86_64 29/32 2022-03-29T17:48:02.516 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:17.1.0-56.g60fdd357.el8.x86_64 29/32 2022-03-29T17:48:02.704 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:17.1.0-56.g60fdd357.el8.x86_64 30/32 2022-03-29T17:48:02.927 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:17.1.0-56.g60fdd357.el8.x86_64 30/32 2022-03-29T17:48:03.096 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:17.1.0-56.g60fdd357.el8.x86_64 31/32 2022-03-29T17:48:03.112 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:17.1.0-56.g60fdd357.el8.x86_64 31/32 2022-03-29T17:48:03.261 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:17.1.0-56.g60fdd357.el8.x86_64 32/32 2022-03-29T17:48:03.655 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-2:17.1.0-56.g60fdd357.el8.x86_64 32/32 2022-03-29T17:48:03.655 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:17.1.0-56.g60fdd357.el8.x86_64 1/32 2022-03-29T17:48:03.656 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:17.1.0-56.g60fdd357.el8.x86_64 2/32 2022-03-29T17:48:03.656 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:17.1.0-56.g60fdd357.el8.x86_64 3/32 2022-03-29T17:48:03.656 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:17.1.0-56.g60fdd357.el8.x86_64 4/32 2022-03-29T17:48:03.656 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:17.1.0-56.g60fdd357.el8.x86_64 5/32 2022-03-29T17:48:03.657 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:17.1.0-56.g60fdd357.el8.no 6/32 2022-03-29T17:48:03.657 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-03-29T17:48:03.657 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-03-29T17:48:03.657 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-03-29T17:48:03.657 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-03-29T17:48:03.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-03-29T17:48:03.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-03-29T17:48:03.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-03-29T17:48:03.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-03-29T17:48:03.659 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-03-29T17:48:03.659 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-03-29T17:48:03.659 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-03-29T17:48:03.659 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-03-29T17:48:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-03-29T17:48:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-03-29T17:48:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-03-29T17:48:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-03-29T17:48:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-03-29T17:48:03.661 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-03-29T17:48:03.661 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-03-29T17:48:03.661 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-03-29T17:48:03.661 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-03-29T17:48:03.662 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-03-29T17:48:03.662 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-03-29T17:48:03.662 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-03-29T17:48:03.662 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-03-29T17:48:05.271 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-03-29T17:48:05.272 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:05.272 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:48:05.272 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:48:05.273 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:48:05.273 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:48:05.273 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:48:05.273 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:48:05.274 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:48:05.274 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-03-29T17:48:05.274 INFO:teuthology.orchestra.run.smithi138.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-03-29T17:48:05.274 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-03-29T17:48:05.275 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-03-29T17:48:05.275 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-03-29T17:48:05.275 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-03-29T17:48:05.275 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-6.2-6.el8.noarch 2022-03-29T17:48:05.276 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-03-29T17:48:05.276 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-03-29T17:48:05.276 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.0.6-13.el8.noarch 2022-03-29T17:48:05.276 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-03-29T17:48:05.277 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-03-29T17:48:05.277 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-03-29T17:48:05.277 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-2.6-1.el8.noarch 2022-03-29T17:48:05.277 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-03-29T17:48:05.278 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.14-14.el8.noarch 2022-03-29T17:48:05.278 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2017.2-9.el8.noarch 2022-03-29T17:48:05.278 INFO:teuthology.orchestra.run.smithi138.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-03-29T17:48:05.278 INFO:teuthology.orchestra.run.smithi138.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-03-29T17:48:05.279 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-03-29T17:48:05.279 INFO:teuthology.orchestra.run.smithi138.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-03-29T17:48:05.279 INFO:teuthology.orchestra.run.smithi138.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-03-29T17:48:05.279 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-03-29T17:48:05.280 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-03-29T17:48:05.280 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-03-29T17:48:05.280 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-03-29T17:48:05.280 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:05.280 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:07.296 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-base 2022-03-29T17:48:07.483 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:07.483 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:07.883 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:00 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:08.690 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-base-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:48:08.742 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:08.743 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:48:08.743 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:08.943 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install cephadm 2022-03-29T17:48:09.120 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:09.120 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:09.522 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:02 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:10.341 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:10.342 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:10.342 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:48:10.343 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:10.343 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:48:10.343 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 71 k 2022-03-29T17:48:10.343 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:10.344 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:48:10.344 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:10.344 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-03-29T17:48:10.344 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:10.344 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 71 k 2022-03-29T17:48:10.345 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 318 k 2022-03-29T17:48:10.345 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:48:11.156 INFO:teuthology.orchestra.run.smithi138.stdout:cephadm-17.1.0-56.g60fdd357.el8.noarch.rpm 87 kB/s | 71 kB 00:00 2022-03-29T17:48:11.199 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:48:11.199 INFO:teuthology.orchestra.run.smithi138.stdout:Total 83 kB/s | 71 kB 00:00 2022-03-29T17:48:11.199 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:48:11.205 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:48:11.205 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:48:11.207 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:48:11.211 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:48:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:48:21.820 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:17.1.0-56.g60fdd357.el8.noarch 1/1 2022-03-29T17:48:23.036 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:17.1.0-56.g60fdd357.el8.noarch 1/1 2022-03-29T17:48:23.253 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:17.1.0-56.g60fdd357.el8.noarch 1/1 2022-03-29T17:48:23.640 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:17.1.0-56.g60fdd357.el8.noarch 1/1 2022-03-29T17:48:23.640 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:23.641 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:48:23.641 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:48:23.641 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:23.641 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:23.850 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-immutable-object-cache 2022-03-29T17:48:24.040 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:24.041 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:24.447 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:17 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:25.303 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:25.305 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:25.305 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2022-03-29T17:48:25.306 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:25.307 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:48:25.307 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:17.1.0-56.g60fdd357.el8 ceph 155 k 2022-03-29T17:48:25.307 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:25.308 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:48:25.308 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:25.308 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-03-29T17:48:25.309 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:25.309 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 155 k 2022-03-29T17:48:25.309 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 534 k 2022-03-29T17:48:25.310 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:48:25.509 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-immutable-object-cache-17.1.0-56.g60fdd357 764 kB/s | 155 kB 00:00 2022-03-29T17:48:25.512 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:48:25.512 INFO:teuthology.orchestra.run.smithi138.stdout:Total 760 kB/s | 155 kB 00:00 2022-03-29T17:48:25.513 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:48:25.520 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:48:25.521 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:48:25.568 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:48:25.571 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:48:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:48:25.867 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:17.1.0-56.g60fdd357.el 1/1 2022-03-29T17:48:26.244 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.1.0-56.g60fdd357.el 1/1 2022-03-29T17:48:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:17.1.0-56.g60fdd357.el 1/1 2022-03-29T17:48:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:26.599 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:48:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:48:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:26.600 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:26.762 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr 2022-03-29T17:48:26.943 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:26.944 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:27.344 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:20 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:28.153 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-mgr-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:48:28.206 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:28.207 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:48:28.207 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:28.280 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-dashboard 2022-03-29T17:48:28.458 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:28.459 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:28.859 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:21 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:29.673 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:29.674 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:48:29.675 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:29.675 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:48:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 1.6 M 2022-03-29T17:48:29.675 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-03-29T17:48:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 23 k 2022-03-29T17:48:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 15 k 2022-03-29T17:48:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2022-03-29T17:48:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-03-29T17:48:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-03-29T17:48:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:29.677 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:48:29.677 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:29.678 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2022-03-29T17:48:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:29.679 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 1.9 M 2022-03-29T17:48:29.680 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 7.5 M 2022-03-29T17:48:29.680 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:48:29.847 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): ceph-grafana-dashboards-17.1.0-56.g60fdd 133 kB/s | 23 kB 00:00 2022-03-29T17:48:29.864 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): ceph-prometheus-alerts-17.1.0-56.g60fdd3 78 kB/s | 15 kB 00:00 2022-03-29T17:48:29.906 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 729 kB/s | 43 kB 00:00 2022-03-29T17:48:29.965 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 335 kB/s | 34 kB 00:00 2022-03-29T17:48:30.048 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): ceph-mgr-dashboard-17.1.0-56.g60fdd357.e 4.4 MB/s | 1.6 MB 00:00 2022-03-29T17:48:30.098 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2022-03-29T17:48:30.100 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:48:30.101 INFO:teuthology.orchestra.run.smithi138.stdout:Total 4.5 MB/s | 1.9 MB 00:00 2022-03-29T17:48:30.110 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:48:30.118 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:48:30.120 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:48:30.198 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:48:30.201 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:48:30.475 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:48:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-03-29T17:48:30.900 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-03-29T17:48:31.110 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-03-29T17:48:31.239 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:17.1.0-56.g60fdd357.el8.noa 4/6 2022-03-29T17:48:31.593 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:17.1.0-56.g60fdd357.el8.no 5/6 2022-03-29T17:48:31.821 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:17.1.0-56.g60fdd357.el8.noarch 6/6 2022-03-29T17:48:32.070 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.1.0-56.g60fdd357.el8.noarch 6/6 2022-03-29T17:48:32.071 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:17.1.0-56.g60fdd357.el8.no 1/6 2022-03-29T17:48:32.071 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:17.1.0-56.g60fdd357.el8.noarch 2/6 2022-03-29T17:48:32.071 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:17.1.0-56.g60fdd357.el8.noa 3/6 2022-03-29T17:48:32.071 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-03-29T17:48:32.071 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-03-29T17:48:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-03-29T17:48:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:32.483 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:48:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:48:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:48:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:48:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-03-29T17:48:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-03-29T17:48:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.4.1-12.el8.noarch 2022-03-29T17:48:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:32.486 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:32.585 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-diskprediction-local 2022-03-29T17:48:32.763 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:32.763 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:33.165 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:26 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:33.991 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:33.992 INFO:teuthology.orchestra.run.smithi138.stdout:===================================================================================================== 2022-03-29T17:48:33.993 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:48:33.993 INFO:teuthology.orchestra.run.smithi138.stdout:===================================================================================================== 2022-03-29T17:48:33.993 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:48:33.993 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 7.4 M 2022-03-29T17:48:33.993 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-03-29T17:48:33.994 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 8.5.0-10.el8 CentOS-Base 643 k 2022-03-29T17:48:33.994 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 8.5.0-10.el8 CentOS-Base 171 k 2022-03-29T17:48:33.994 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2022-03-29T17:48:33.994 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2022-03-29T17:48:33.995 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-03-29T17:48:33.995 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-03-29T17:48:33.995 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-03-29T17:48:33.995 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:33.996 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:48:33.996 INFO:teuthology.orchestra.run.smithi138.stdout:===================================================================================================== 2022-03-29T17:48:33.996 INFO:teuthology.orchestra.run.smithi138.stdout:Install 8 Packages 2022-03-29T17:48:33.996 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:33.997 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 35 M 2022-03-29T17:48:33.997 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 210 M 2022-03-29T17:48:33.998 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:48:34.190 INFO:teuthology.orchestra.run.smithi138.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2022-03-29T17:48:34.315 INFO:teuthology.orchestra.run.smithi138.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2022-03-29T17:48:34.349 INFO:teuthology.orchestra.run.smithi138.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2022-03-29T17:48:34.499 INFO:teuthology.orchestra.run.smithi138.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 9.6 MB/s | 4.9 MB 00:00 2022-03-29T17:48:34.558 INFO:teuthology.orchestra.run.smithi138.stdout:(5/8): libgfortran-8.5.0-10.el8.x86_64.rpm 11 MB/s | 643 kB 00:00 2022-03-29T17:48:34.600 INFO:teuthology.orchestra.run.smithi138.stdout:(6/8): libquadmath-8.5.0-10.el8.x86_64.rpm 4.0 MB/s | 171 kB 00:00 2022-03-29T17:48:34.775 INFO:teuthology.orchestra.run.smithi138.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 32 MB/s | 14 MB 00:00 2022-03-29T17:48:34.951 INFO:teuthology.orchestra.run.smithi138.stdout:(8/8): ceph-mgr-diskprediction-local-17.1.0-56. 7.7 MB/s | 7.4 MB 00:00 2022-03-29T17:48:34.953 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:48:34.953 INFO:teuthology.orchestra.run.smithi138.stdout:Total 37 MB/s | 35 MB 00:00 2022-03-29T17:48:35.183 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:48:35.201 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:48:35.202 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:48:35.574 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:48:35.577 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:48:36.164 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:48:36.315 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-8.5.0-10.el8.x86_64 1/8 2022-03-29T17:48:36.453 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libquadmath-8.5.0-10.el8.x86_64 1/8 2022-03-29T17:48:36.657 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-8.5.0-10.el8.x86_64 2/8 2022-03-29T17:48:37.263 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libgfortran-8.5.0-10.el8.x86_64 2/8 2022-03-29T17:48:37.498 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2022-03-29T17:48:38.112 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2022-03-29T17:48:38.282 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-03-29T17:48:38.967 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-03-29T17:48:39.318 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-03-29T17:48:39.539 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-03-29T17:48:41.297 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-03-29T17:48:42.552 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-03-29T17:48:42.772 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:17.1.0-56.g60fdd357. 8/8 2022-03-29T17:48:43.056 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.1.0-56.g60fdd357. 8/8 2022-03-29T17:48:43.057 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.1.0-56.g60fdd357. 1/8 2022-03-29T17:48:43.057 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2022-03-29T17:48:43.057 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2022-03-29T17:48:43.057 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-03-29T17:48:43.058 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-03-29T17:48:43.058 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-03-29T17:48:43.058 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-8.5.0-10.el8.x86_64 7/8 2022-03-29T17:48:43.473 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-8.5.0-10.el8.x86_64 8/8 2022-03-29T17:48:43.473 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:43.474 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:48:43.474 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:48:43.474 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-8.5.0-10.el8.x86_64 2022-03-29T17:48:43.475 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-8.5.0-10.el8.x86_64 2022-03-29T17:48:43.475 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.15-3.el8.x86_64 2022-03-29T17:48:43.475 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-03-29T17:48:43.475 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-03-29T17:48:43.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-03-29T17:48:43.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-03-29T17:48:43.476 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:43.476 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:43.575 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-rook 2022-03-29T17:48:43.751 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:43.752 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:44.153 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:37 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:45.010 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:45.012 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2022-03-29T17:48:45.012 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:48:45.012 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2022-03-29T17:48:45.012 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:48:45.013 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 47 k 2022-03-29T17:48:45.013 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-03-29T17:48:45.013 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-03-29T17:48:45.013 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-03-29T17:48:45.014 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-03-29T17:48:45.014 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-03-29T17:48:45.015 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-03-29T17:48:45.015 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-03-29T17:48:45.015 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2022-03-29T17:48:45.015 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-03-29T17:48:45.016 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-03-29T17:48:45.016 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2022-03-29T17:48:45.016 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-03-29T17:48:45.017 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-03-29T17:48:45.017 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:45.017 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:48:45.018 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2022-03-29T17:48:45.018 INFO:teuthology.orchestra.run.smithi138.stdout:Install 13 Packages 2022-03-29T17:48:45.019 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:45.020 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 1.9 M 2022-03-29T17:48:45.021 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 25 M 2022-03-29T17:48:45.021 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:48:45.079 INFO:teuthology.orchestra.run.smithi138.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 416 kB/s | 27 kB 00:00 2022-03-29T17:48:45.105 INFO:teuthology.orchestra.run.smithi138.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 224 kB/s | 20 kB 00:00 2022-03-29T17:48:45.130 INFO:teuthology.orchestra.run.smithi138.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.7 MB/s | 126 kB 00:00 2022-03-29T17:48:45.155 INFO:teuthology.orchestra.run.smithi138.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-03-29T17:48:45.181 INFO:teuthology.orchestra.run.smithi138.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2022-03-29T17:48:45.198 INFO:teuthology.orchestra.run.smithi138.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2022-03-29T17:48:45.315 INFO:teuthology.orchestra.run.smithi138.stdout:(7/13): ceph-mgr-rook-17.1.0-56.g60fdd357.el8.n 157 kB/s | 47 kB 00:00 2022-03-29T17:48:45.367 INFO:teuthology.orchestra.run.smithi138.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 97 kB/s | 16 kB 00:00 2022-03-29T17:48:45.390 INFO:teuthology.orchestra.run.smithi138.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 157 kB/s | 33 kB 00:00 2022-03-29T17:48:45.507 INFO:teuthology.orchestra.run.smithi138.stdout:(10/13): python3-rsa-4.8-1.el8.noarch.rpm 527 kB/s | 61 kB 00:00 2022-03-29T17:48:45.533 INFO:teuthology.orchestra.run.smithi138.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 373 kB/s | 81 kB 00:00 2022-03-29T17:48:45.608 INFO:teuthology.orchestra.run.smithi138.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 610 kB/s | 61 kB 00:00 2022-03-29T17:48:46.158 INFO:teuthology.orchestra.run.smithi138.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.5 MB/s | 1.2 MB 00:00 2022-03-29T17:48:46.160 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:48:46.160 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2022-03-29T17:48:46.214 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:48:46.230 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:48:46.230 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:48:46.309 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:48:46.312 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:48:46.811 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:48:47.024 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-03-29T17:48:47.238 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-03-29T17:48:47.473 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-03-29T17:48:47.736 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-03-29T17:48:47.952 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-03-29T17:48:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-03-29T17:48:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-03-29T17:48:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-03-29T17:48:49.825 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-03-29T17:48:50.128 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-03-29T17:48:50.328 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-03-29T17:48:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-03-29T17:48:50.796 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:17.1.0-56.g60fdd357.el8.noarch 13/13 2022-03-29T17:48:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:17.1.0-56.g60fdd357.el8.noarch 13/13 2022-03-29T17:48:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:17.1.0-56.g60fdd357.el8.noarch 1/13 2022-03-29T17:48:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-03-29T17:48:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-03-29T17:48:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-03-29T17:48:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-03-29T17:48:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-03-29T17:48:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-03-29T17:48:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-03-29T17:48:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-03-29T17:48:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-03-29T17:48:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-03-29T17:48:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-03-29T17:48:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-03-29T17:48:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:51.539 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:48:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:48:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-03-29T17:48:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-03-29T17:48:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-03-29T17:48:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-03-29T17:48:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-03-29T17:48:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-03-29T17:48:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-03-29T17:48:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-03-29T17:48:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-03-29T17:48:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-03-29T17:48:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.8-1.el8.noarch 2022-03-29T17:48:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-03-29T17:48:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:51.542 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:51.638 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-cephadm 2022-03-29T17:48:51.816 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:51.816 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:52.221 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:45 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:53.045 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:53.046 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:48:53.047 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:53.047 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:48:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 103 k 2022-03-29T17:48:53.047 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-03-29T17:48:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-03-29T17:48:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-03-29T17:48:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-03-29T17:48:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:53.048 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:48:53.049 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:53.049 INFO:teuthology.orchestra.run.smithi138.stdout:Install 4 Packages 2022-03-29T17:48:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:53.050 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 5.9 M 2022-03-29T17:48:53.050 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 25 M 2022-03-29T17:48:53.050 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:48:53.138 INFO:teuthology.orchestra.run.smithi138.stdout:(1/4): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.0 MB/s | 538 kB 00:00 2022-03-29T17:48:53.262 INFO:teuthology.orchestra.run.smithi138.stdout:(2/4): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2022-03-29T17:48:53.537 INFO:teuthology.orchestra.run.smithi138.stdout:(3/4): ceph-mgr-cephadm-17.1.0-56.g60fdd357.el8 210 kB/s | 103 kB 00:00 2022-03-29T17:48:53.888 INFO:teuthology.orchestra.run.smithi138.stdout:(4/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 638 kB/s | 476 kB 00:00 2022-03-29T17:48:53.890 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:48:53.891 INFO:teuthology.orchestra.run.smithi138.stdout:Total 7.0 MB/s | 5.9 MB 00:00 2022-03-29T17:48:53.942 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:48:53.951 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:48:53.952 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:48:54.052 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:48:54.054 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:48:54.371 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:48:55.555 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 1/4 2022-03-29T17:48:55.947 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 2/4 2022-03-29T17:48:56.185 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 3/4 2022-03-29T17:48:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:17.1.0-56.g60fdd357.el8.noarch 4/4 2022-03-29T17:48:56.690 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.1.0-56.g60fdd357.el8.noarch 4/4 2022-03-29T17:48:56.690 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:17.1.0-56.g60fdd357.el8.noarch 1/4 2022-03-29T17:48:56.690 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/4 2022-03-29T17:48:56.691 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/4 2022-03-29T17:48:57.062 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/4 2022-03-29T17:48:57.063 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:57.063 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:48:57.064 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:48:57.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-03-29T17:48:57.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.5.1-7.el8.noarch 2022-03-29T17:48:57.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-03-29T17:48:57.065 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:57.065 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:48:57.161 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-fuse 2022-03-29T17:48:57.339 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:57.340 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:48:57.741 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:50 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:48:58.565 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:48:58.566 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:58.566 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:48:58.567 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:58.567 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:48:58.567 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse x86_64 2:17.1.0-56.g60fdd357.el8 ceph 859 k 2022-03-29T17:48:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-03-29T17:48:58.568 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2022-03-29T17:48:58.568 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:48:58.569 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:48:58.569 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2022-03-29T17:48:58.569 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:48:58.569 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 942 k 2022-03-29T17:48:58.570 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 3.4 M 2022-03-29T17:48:58.570 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:48:58.638 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.1 MB/s | 82 kB 00:00 2022-03-29T17:48:59.105 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-fuse-17.1.0-56.g60fdd357.el8.x86_64 1.6 MB/s | 859 kB 00:00 2022-03-29T17:48:59.106 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:48:59.106 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.7 MB/s | 942 kB 00:00 2022-03-29T17:48:59.116 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:48:59.137 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:48:59.137 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:48:59.218 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:48:59.220 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:48:59.383 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:48:59.682 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2022-03-29T17:48:59.903 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:17.1.0-56.g60fdd357.el8.x86_64 2/2 2022-03-29T17:49:00.276 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-fuse-2:17.1.0-56.g60fdd357.el8.x86_64 2/2 2022-03-29T17:49:00.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:17.1.0-56.g60fdd357.el8.x86_64 1/2 2022-03-29T17:49:00.887 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2022-03-29T17:49:00.888 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:00.888 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:49:00.888 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:17.1.0-56.g60fdd357.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2022-03-29T17:49:00.888 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:00.889 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:01.054 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-volume 2022-03-29T17:49:01.236 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:01.237 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:01.649 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:54 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:02.465 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:02.466 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-03-29T17:49:02.467 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:02.467 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:49:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:17.1.0-56.g60fdd357.el8 ceph-noarch 257 k 2022-03-29T17:49:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:02.467 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:49:02.468 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:02.468 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-03-29T17:49:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:02.468 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 257 k 2022-03-29T17:49:02.469 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 1.3 M 2022-03-29T17:49:02.469 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:49:02.953 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-volume-17.1.0-56.g60fdd357.el8.noarch.rpm 529 kB/s | 257 kB 00:00 2022-03-29T17:49:02.954 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:49:02.954 INFO:teuthology.orchestra.run.smithi138.stdout:Total 527 kB/s | 257 kB 00:00 2022-03-29T17:49:02.955 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:49:02.964 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:49:02.965 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:49:02.985 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:49:02.988 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:49:03.155 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:49:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:17.1.0-56.g60fdd357.el8.noarch 1/1 2022-03-29T17:49:03.831 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:17.1.0-56.g60fdd357.el8.noarch 1/1 2022-03-29T17:49:04.192 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:17.1.0-56.g60fdd357.el8.noarch 1/1 2022-03-29T17:49:04.193 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:04.193 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:49:04.193 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:17.1.0-56.g60fdd357.el8.noarch 2022-03-29T17:49:04.193 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:04.194 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:04.407 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados-devel 2022-03-29T17:49:04.590 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:04.590 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:05.004 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:03:58 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:05.817 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:05.818 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:05.818 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2022-03-29T17:49:05.819 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:05.819 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:49:05.819 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:17.1.0-56.g60fdd357.el8 ceph 127 k 2022-03-29T17:49:05.819 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:05.819 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:49:05.820 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:05.820 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-03-29T17:49:05.820 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:05.820 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 127 k 2022-03-29T17:49:05.821 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 448 k 2022-03-29T17:49:05.821 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:49:06.194 INFO:teuthology.orchestra.run.smithi138.stdout:librados-devel-17.1.0-56.g60fdd357.el8.x86_64.r 339 kB/s | 127 kB 00:00 2022-03-29T17:49:06.195 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:49:06.196 INFO:teuthology.orchestra.run.smithi138.stdout:Total 338 kB/s | 127 kB 00:00 2022-03-29T17:49:06.196 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:49:06.201 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:49:06.201 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:49:06.250 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:49:06.253 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:49:06.361 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:49:06.567 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados-devel-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:07.209 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:07.210 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:07.210 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:49:07.210 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:49:07.210 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:07.211 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:07.440 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs2 2022-03-29T17:49:07.619 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:07.620 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:08.037 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:01 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:08.813 INFO:teuthology.orchestra.run.smithi138.stdout:Package libcephfs2-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:49:08.866 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:08.867 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:49:08.867 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:08.953 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs-devel 2022-03-29T17:49:09.129 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:09.129 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:09.534 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:02 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:10.393 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:10.394 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2022-03-29T17:49:10.394 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:10.395 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:49:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:17.1.0-56.g60fdd357.el8 ceph 24 k 2022-03-29T17:49:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:10.395 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:49:10.396 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:10.396 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-03-29T17:49:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:10.396 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 24 k 2022-03-29T17:49:10.396 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 106 k 2022-03-29T17:49:10.397 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:49:10.966 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs-devel-17.1.0-56.g60fdd357.el8.x86_64. 43 kB/s | 24 kB 00:00 2022-03-29T17:49:10.967 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:49:10.967 INFO:teuthology.orchestra.run.smithi138.stdout:Total 43 kB/s | 24 kB 00:00 2022-03-29T17:49:10.967 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:49:10.971 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:49:10.972 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:49:10.974 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:49:10.977 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:49:11.366 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:49:12.667 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs-devel-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:13.266 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:13.266 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:13.267 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:49:13.267 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:49:13.267 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:13.267 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:13.356 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados2 2022-03-29T17:49:13.533 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:13.533 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:13.938 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:06 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:14.696 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:49:14.750 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:14.751 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:49:14.751 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:14.835 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librbd1 2022-03-29T17:49:15.013 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:15.013 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:15.415 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:08 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:16.181 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:49:16.234 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:16.235 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:49:16.235 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:16.314 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rados 2022-03-29T17:49:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:16.491 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:16.895 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:09 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:17.678 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rados-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:49:17.732 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:17.733 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:49:17.733 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:17.805 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rgw 2022-03-29T17:49:17.982 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:17.982 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:18.386 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:11 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:19.153 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rgw-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:49:19.209 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:19.210 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:49:19.211 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:19.292 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-cephfs 2022-03-29T17:49:19.469 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:19.470 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:19.868 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:12 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:20.632 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-cephfs-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:49:20.686 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:20.691 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:49:20.692 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:20.767 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rbd 2022-03-29T17:49:20.944 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:20.945 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:21.350 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:14 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:22.119 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rbd-2:17.1.0-56.g60fdd357.el8.x86_64 is already installed. 2022-03-29T17:49:22.172 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:22.173 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-03-29T17:49:22.173 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:22.259 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-fuse 2022-03-29T17:49:22.436 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:22.437 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:22.846 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:15 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:23.668 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:23.669 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2022-03-29T17:49:23.669 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:23.670 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:49:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:17.1.0-56.g60fdd357.el8 ceph 87 k 2022-03-29T17:49:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:23.670 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:49:23.671 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:23.671 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-03-29T17:49:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:23.671 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 87 k 2022-03-29T17:49:23.671 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 280 k 2022-03-29T17:49:23.672 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:49:23.944 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-fuse-17.1.0-56.g60fdd357.el8.x86_64.rpm 319 kB/s | 87 kB 00:00 2022-03-29T17:49:23.945 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:49:23.945 INFO:teuthology.orchestra.run.smithi138.stdout:Total 317 kB/s | 87 kB 00:00 2022-03-29T17:49:23.945 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:49:23.954 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:49:23.954 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:49:24.003 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:49:24.004 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:49:24.121 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:49:24.352 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:24.633 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-fuse-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:25.026 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:49:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:49:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:25.027 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:25.208 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-mirror 2022-03-29T17:49:25.386 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:25.387 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:25.798 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:18 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:26.609 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:26.610 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:26.610 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2022-03-29T17:49:26.610 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:26.611 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:49:26.611 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:17.1.0-56.g60fdd357.el8 ceph 3.4 M 2022-03-29T17:49:26.611 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:26.611 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:49:26.611 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:26.612 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-03-29T17:49:26.612 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:26.612 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.4 M 2022-03-29T17:49:26.612 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 18 M 2022-03-29T17:49:26.613 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:49:27.941 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-mirror-17.1.0-56.g60fdd357.el8.x86_64.rpm 2.6 MB/s | 3.4 MB 00:01 2022-03-29T17:49:27.942 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:49:27.942 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.6 MB/s | 3.4 MB 00:01 2022-03-29T17:49:27.943 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:49:27.953 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:49:27.954 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:49:28.014 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:49:28.017 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:49:28.464 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:49:28.790 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:29.178 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:29.539 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:29.540 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:29.540 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:49:29.540 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:49:29.540 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:29.541 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:29.729 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-nbd 2022-03-29T17:49:29.910 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:29.911 INFO:teuthology.orchestra.run.smithi138.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-29T17:49:30.325 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:04:23 ago on Tue 29 Mar 2022 05:45:07 PM UTC. 2022-03-29T17:49:31.139 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-03-29T17:49:31.140 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:31.140 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2022-03-29T17:49:31.140 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:31.140 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-03-29T17:49:31.141 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:17.1.0-56.g60fdd357.el8 ceph 174 k 2022-03-29T17:49:31.141 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:31.141 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-03-29T17:49:31.142 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-03-29T17:49:31.142 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-03-29T17:49:31.142 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:31.142 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 174 k 2022-03-29T17:49:31.143 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 612 k 2022-03-29T17:49:31.143 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-03-29T17:49:31.349 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-nbd-17.1.0-56.g60fdd357.el8.x86_64.rpm 838 kB/s | 174 kB 00:00 2022-03-29T17:49:31.349 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-03-29T17:49:31.350 INFO:teuthology.orchestra.run.smithi138.stdout:Total 833 kB/s | 174 kB 00:00 2022-03-29T17:49:31.350 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-03-29T17:49:31.359 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-03-29T17:49:31.360 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-03-29T17:49:31.408 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-03-29T17:49:31.410 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-03-29T17:49:31.530 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-03-29T17:49:31.707 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:31.960 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-nbd-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:32.351 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:17.1.0-56.g60fdd357.el8.x86_64 1/1 2022-03-29T17:49:32.351 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:32.351 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-03-29T17:49:32.352 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:17.1.0-56.g60fdd357.el8.x86_64 2022-03-29T17:49:32.352 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:32.352 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-03-29T17:49:32.545 DEBUG:teuthology.parallel:result is None 2022-03-29T17:49:32.546 DEBUG:teuthology.orchestra.run.smithi138:> sudo lsb_release -is 2022-03-29T17:49:32.582 INFO:teuthology.orchestra.run.smithi138.stdout:CentOSStream 2022-03-29T17:49:32.584 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-03-29T17:49:32.584 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 2022-03-29T17:49:32.876 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-03-29T17:49:32.898 INFO:teuthology.orchestra.run.smithi138.stdout:17.1.0-56.g60fdd357.el8 2022-03-29T17:49:32.899 INFO:teuthology.packaging:The installed version of ceph is 17.1.0-56.g60fdd357.el8 2022-03-29T17:49:32.899 INFO:teuthology.task.install:The correct ceph version 17.1.0-56.g60fdd357 is installed. 2022-03-29T17:49:32.900 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-29T17:49:32.900 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:32.901 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-29T17:49:32.965 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-29T17:49:32.965 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:32.966 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2022-03-29T17:49:33.031 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-29T17:49:33.096 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-29T17:49:33.097 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:33.097 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-29T17:49:33.164 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-29T17:49:33.230 INFO:teuthology.run_tasks:Running task ceph... 2022-03-29T17:49:33.320 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-03-29T17:49:33.321 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /var/log/ceph 2022-03-29T17:49:33.347 INFO:tasks.ceph:Disabling ceph logrotate... 2022-03-29T17:49:33.348 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/logrotate.d/ceph 2022-03-29T17:49:33.414 INFO:tasks.ceph:Creating extra log directories... 2022-03-29T17:49:33.414 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-03-29T17:49:33.494 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-03-29T17:49:33.495 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-03-29T17:49:33.495 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2022-03-20_01:01:02-crimson-rados-quincy-distro-default-smithi/6747525', 'branch': 'quincy', 'description': 'crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_python_api_tests_old_format}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '6747525', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2022-03-20_01:01:02-crimson-rados-quincy-distro-default-smithi', 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'quincy'}, 'ceph': {'conf': {'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(SLOW_OPS\\)', 'slow request'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'quincy', 'sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 1000, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_60fdd357d1aeaa5f5a03a623dc666183c6bee5d5/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5', 'targets': {'smithi138.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+W4RzCfhLa+nXiuw2c9D/5nDX3hHa2L3Nyp6KyrvAaaVIRfQj1I53bs5iq5Q0S2SbftuTF9kTUzWweNmoiWh5/Z5JpipjO7p3lI0NXrX+mf0AzXVfePQk99eVrIqGx8ryoha0QMmm7jo2TPSD18xpR7gogfSMnX8N7IrfncWrOShipp7Jk81PjCxWAtTVz2ghf4zFFtYl88DaV0EC2xZYVkRC9j8vaF0vh7nLjrvtlPKBW1KAgWFGYzlnKM/e2/IUgVSnlg4EpC4pZcJKh1qqR7wctmg6CLYp+LCTldpt9kR4QOdAKSVB2yMbkOoWwLbPtdfBT0O5CyOzc933E/fZEkUPDQPLyKpTOxCqU6XJjCX0aSj3MzDjSk/sW70et1TXFkQBhrGIVmJXWeKGUDFW8wfeUFpg1J6hvxFAgrc35ZvgOgwWo+GrzGXJrk+BBEucHnjCLipGeebztraFCqciOArJNtPJ1p+je3s9z07nBXAH8NK91YOzStkGt2PI1j0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(SLOW_OPS\\)', 'slow request'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_librbd_python.sh']}}}], 'teuthology_branch': 'master', 'teuthology_sha1': 'a4e18662e5de936eb0ac089e531b933ecbcadb0a', 'timestamp': '2022-03-20_01:01:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.22301'} 2022-03-29T17:49:33.496 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-03-29T17:49:33.539 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/run/ceph 2022-03-29T17:49:33.605 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:33.606 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2022-03-29T17:49:33.662 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-29T17:49:33.663 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-29T17:49:33.719 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-29T17:49:33.719 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-29T17:49:33.719 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 156854 Links: 1 2022-03-29T17:49:33.720 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-29T17:49:33.720 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2022-03-29T17:49:33.720 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-03-29 17:49:28.947468158 +0000 2022-03-29T17:49:33.720 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-03-29 17:44:24.323935781 +0000 2022-03-29T17:49:33.720 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-03-29 17:44:24.323935781 +0000 2022-03-29T17:49:33.721 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-03-29T17:49:33.728 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-03-29T17:49:33.729 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-03-29T17:49:33.729 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000208566 s, 2.5 MB/s 2022-03-29T17:49:33.731 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-29T17:49:33.788 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-29T17:49:33.788 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-29T17:49:33.789 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 157870 Links: 1 2022-03-29T17:49:33.789 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-29T17:49:33.789 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2022-03-29T17:49:33.789 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-03-29 17:49:28.947468158 +0000 2022-03-29T17:49:33.789 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-03-29 17:44:24.655929087 +0000 2022-03-29T17:49:33.790 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-03-29 17:44:24.655929087 +0000 2022-03-29T17:49:33.790 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-03-29T17:49:33.797 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-03-29T17:49:33.797 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-03-29T17:49:33.798 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000209893 s, 2.4 MB/s 2022-03-29T17:49:33.800 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-29T17:49:33.857 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-29T17:49:33.857 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-29T17:49:33.858 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 156903 Links: 1 2022-03-29T17:49:33.858 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-29T17:49:33.858 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2022-03-29T17:49:33.858 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-03-29 17:49:28.947468158 +0000 2022-03-29T17:49:33.859 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-03-29 17:44:25.002922092 +0000 2022-03-29T17:49:33.859 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-03-29 17:44:25.002922092 +0000 2022-03-29T17:49:33.859 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-03-29T17:49:33.866 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-03-29T17:49:33.869 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-03-29T17:49:33.869 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.0001918 s, 2.7 MB/s 2022-03-29T17:49:33.870 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-29T17:49:33.928 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-29T17:49:33.928 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-29T17:49:33.928 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 157925 Links: 1 2022-03-29T17:49:33.929 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-29T17:49:33.929 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2022-03-29T17:49:33.929 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-03-29 17:49:28.947468158 +0000 2022-03-29T17:49:33.929 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-03-29 17:44:25.341915257 +0000 2022-03-29T17:49:33.930 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-03-29 17:44:25.341915257 +0000 2022-03-29T17:49:33.930 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-03-29T17:49:33.937 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-03-29T17:49:33.938 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-03-29T17:49:33.938 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000232685 s, 2.2 MB/s 2022-03-29T17:49:33.940 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2022-03-29T17:49:33.940 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2022-03-29T17:49:33.941 INFO:tasks.ceph:Generating config... 2022-03-29T17:49:33.942 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2022-03-29T17:49:33.943 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2022-03-29T17:49:33.943 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2022-03-29T17:49:33.943 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2022-03-29T17:49:33.944 INFO:tasks.ceph:[mon] paxos service trim min = 10 2022-03-29T17:49:33.944 INFO:tasks.ceph:[mon] debug mon = 20 2022-03-29T17:49:33.944 INFO:tasks.ceph:[mon] debug ms = 1 2022-03-29T17:49:33.944 INFO:tasks.ceph:[mon] debug paxos = 20 2022-03-29T17:49:33.945 INFO:tasks.ceph:[osd] debug monc = 20 2022-03-29T17:49:33.945 INFO:tasks.ceph:[osd] debug ms = 1 2022-03-29T17:49:33.945 INFO:tasks.ceph:[osd] debug osd = 20 2022-03-29T17:49:33.946 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2022-03-29T17:49:33.946 INFO:tasks.ceph:[global] ms client mode = crc 2022-03-29T17:49:33.946 INFO:tasks.ceph:[global] ms cluster mode = crc 2022-03-29T17:49:33.947 INFO:tasks.ceph:[global] ms mon client mode = crc 2022-03-29T17:49:33.947 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2022-03-29T17:49:33.947 INFO:tasks.ceph:[global] ms mon service mode = crc 2022-03-29T17:49:33.948 INFO:tasks.ceph:[global] ms service mode = crc 2022-03-29T17:49:33.948 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-03-29T17:49:33.948 INFO:tasks.ceph:[mgr] debug ms = 1 2022-03-29T17:49:33.949 INFO:tasks.ceph:Setting up mon.a... 2022-03-29T17:49:33.949 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-03-29T17:49:34.069 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/ceph.keyring 2022-03-29T17:49:34.071 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-03-29T17:49:34.124 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-03-29T17:49:34.150 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.138')] 2022-03-29T17:49:34.150 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.138', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2022-03-29T17:49:34.152 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:34.152 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-03-29T17:49:34.207 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.138 --print /home/ubuntu/cephtest/ceph.monmap 2022-03-29T17:49:34.297 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-03-29T17:49:34.298 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: generated fsid 3be68405-f255-4011-9a3b-ba646329d493 2022-03-29T17:49:34.298 INFO:teuthology.orchestra.run.smithi138.stdout:setting min_mon_release = octopus 2022-03-29T17:49:34.298 INFO:teuthology.orchestra.run.smithi138.stdout:epoch 0 2022-03-29T17:49:34.299 INFO:teuthology.orchestra.run.smithi138.stdout:fsid 3be68405-f255-4011-9a3b-ba646329d493 2022-03-29T17:49:34.299 INFO:teuthology.orchestra.run.smithi138.stdout:last_changed 2022-03-29T17:49:34.296195+0000 2022-03-29T17:49:34.299 INFO:teuthology.orchestra.run.smithi138.stdout:created 2022-03-29T17:49:34.296195+0000 2022-03-29T17:49:34.299 INFO:teuthology.orchestra.run.smithi138.stdout:min_mon_release 15 (octopus) 2022-03-29T17:49:34.300 INFO:teuthology.orchestra.run.smithi138.stdout:election_strategy: 1 2022-03-29T17:49:34.300 INFO:teuthology.orchestra.run.smithi138.stdout:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-03-29T17:49:34.300 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2022-03-29T17:49:34.302 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-03-29T17:49:34.320 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3be68405-f255-4011-9a3b-ba646329d493... 2022-03-29T17:49:34.321 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-03-29T17:49:34.404 INFO:teuthology.orchestra.run.smithi138.stdout:[global] 2022-03-29T17:49:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: chdir = "" 2022-03-29T17:49:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-03-29T17:49:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: auth supported = cephx 2022-03-29T17:49:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: filestore xattr use omap = true 2022-03-29T17:49:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: mon clock drift allowed = 1.000 2022-03-29T17:49:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: osd crush chooseleaf type = 0 2022-03-29T17:49:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: auth debug = true 2022-03-29T17:49:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on old message = true 2022-03-29T17:49:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on bug = true 2022-03-29T17:49:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: mon max pg per osd = 10000 # >= luminous 2022-03-29T17:49:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: mon pg warn max object skew = 0 2022-03-29T17:49:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: # disable pg_autoscaler by default for new pools 2022-03-29T17:49:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_pg_autoscale_mode = off 2022-03-29T17:49:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default size = 2 2022-03-29T17:49:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow primary affinity = true 2022-03-29T17:49:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow pg remap = true 2022-03-29T17:49:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on legacy crush tunables = false 2022-03-29T17:49:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on crush straw calc version zero = false 2022-03-29T17:49:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on no sortbitwise = false 2022-03-29T17:49:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on osd down out interval zero = false 2022-03-29T17:49:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on too few osds = false 2022-03-29T17:49:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-03-29T17:49:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_no_redundancy = false 2022-03-29T17:49:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: mon_allow_pool_size_one = true 2022-03-29T17:49:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-03-29T17:49:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: osd default data pool replay window = 5 2022-03-29T17:49:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: mon allow pool delete = true 2022-03-29T17:49:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: mon cluster log file level = debug 2022-03-29T17:49:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: debug asserts on shutdown = true 2022-03-29T17:49:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: mon health detail to clog = false 2022-03-29T17:49:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: mon host = 172.21.15.138 2022-03-29T17:49:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: ms client mode = crc 2022-03-29T17:49:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: ms cluster mode = crc 2022-03-29T17:49:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: ms mon client mode = crc 2022-03-29T17:49:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: ms mon cluster mode = crc 2022-03-29T17:49:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: ms mon service mode = crc 2022-03-29T17:49:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: ms service mode = crc 2022-03-29T17:49:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: fsid = 3be68405-f255-4011-9a3b-ba646329d493 2022-03-29T17:49:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.416 INFO:teuthology.orchestra.run.smithi138.stdout:[osd] 2022-03-29T17:49:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: osd journal size = 100 2022-03-29T17:49:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub load threshold = 5.0 2022-03-29T17:49:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub max interval = 600 2022-03-29T17:49:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: osd recover clone overlap = true 2022-03-29T17:49:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: osd recovery max chunk = 1048576 2022-03-29T17:49:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug shutdown = true 2022-03-29T17:49:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug op order = true 2022-03-29T17:49:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify stray on activate = true 2022-03-29T17:49:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd open classes on start = true 2022-03-29T17:49:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug pg log writeout = true 2022-03-29T17:49:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd deep scrub update digest min age = 30 2022-03-29T17:49:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd map max advance = 10 2022-03-29T17:49:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: journal zero on create = true 2022-03-29T17:49:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.421 INFO:teuthology.orchestra.run.smithi138.stdout: filestore ondisk finisher threads = 3 2022-03-29T17:49:34.421 INFO:teuthology.orchestra.run.smithi138.stdout: filestore apply finisher threads = 3 2022-03-29T17:49:34.421 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.421 INFO:teuthology.orchestra.run.smithi138.stdout: bdev debug aio = true 2022-03-29T17:49:34.422 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug misdirected ops = true 2022-03-29T17:49:34.422 INFO:teuthology.orchestra.run.smithi138.stdout: debug monc = 20 2022-03-29T17:49:34.422 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-03-29T17:49:34.422 INFO:teuthology.orchestra.run.smithi138.stdout: debug osd = 20 2022-03-29T17:49:34.422 INFO:teuthology.orchestra.run.smithi138.stdout: osd shutdown pgref assert = True 2022-03-29T17:49:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.423 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr] 2022-03-29T17:49:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-03-29T17:49:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: debug mgr = 20 2022-03-29T17:49:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2022-03-29T17:49:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2022-03-29T17:49:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2022-03-29T17:49:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2022-03-29T17:49:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: mgr/telemetry/nag = false 2022-03-29T17:49:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.425 INFO:teuthology.orchestra.run.smithi138.stdout:[mon] 2022-03-29T17:49:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-03-29T17:49:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2022-03-29T17:49:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: debug paxos = 20 2022-03-29T17:49:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2022-03-29T17:49:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: mon data avail warn = 5 2022-03-29T17:49:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: mon mgr mkfs grace = 240 2022-03-29T17:49:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2022-03-29T17:49:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd reporter subtree level = osd 2022-03-29T17:49:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd prime pg temp = true 2022-03-29T17:49:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2022-03-29T17:49:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-03-29T17:49:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: auth mon ticket ttl = 660 # 11m 2022-03-29T17:49:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: auth service ticket ttl = 240 # 4m 2022-03-29T17:49:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: # don't complain about insecure global_id in the test suite 2022-03-29T17:49:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-03-29T17:49:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-03-29T17:49:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: # 1m isn't quite enough 2022-03-29T17:49:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: mon_down_mkfs_grace = 2m 2022-03-29T17:49:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_filestore_osds = false 2022-03-29T17:49:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: mon min osdmap epochs = 50 2022-03-29T17:49:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune interval = 2 2022-03-29T17:49:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune min = 15 2022-03-29T17:49:34.431 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune txsize = 2 2022-03-29T17:49:34.431 INFO:teuthology.orchestra.run.smithi138.stdout: paxos service trim min = 10 2022-03-29T17:49:34.431 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:34.431 INFO:teuthology.orchestra.run.smithi138.stdout:[client] 2022-03-29T17:49:34.432 INFO:teuthology.orchestra.run.smithi138.stdout: rgw cache enabled = true 2022-03-29T17:49:34.432 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable ops log = true 2022-03-29T17:49:34.432 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable usage log = true 2022-03-29T17:49:34.432 INFO:teuthology.orchestra.run.smithi138.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-03-29T17:49:34.432 INFO:teuthology.orchestra.run.smithi138.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-03-29T17:49:34.433 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.a] 2022-03-29T17:49:34.434 INFO:tasks.ceph:Creating admin key on mon.a... 2022-03-29T17:49:34.434 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-03-29T17:49:34.501 INFO:tasks.ceph:Copying monmap to all nodes... 2022-03-29T17:49:34.501 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:34.501 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2022-03-29T17:49:34.518 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:34.519 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2022-03-29T17:49:34.575 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi138.front.sepia.ceph.com 2022-03-29T17:49:34.631 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:34.631 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.keyring 2022-03-29T17:49:34.631 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-03-29T17:49:34.667 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:34.668 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-03-29T17:49:34.725 INFO:tasks.ceph:Setting up mon nodes... 2022-03-29T17:49:34.725 INFO:tasks.ceph:Setting up mgr nodes... 2022-03-29T17:49:34.727 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-03-29T17:49:34.824 INFO:teuthology.orchestra.run.smithi138.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-03-29T17:49:34.827 INFO:tasks.ceph:Setting up mds nodes... 2022-03-29T17:49:34.827 INFO:tasks.ceph_client:Setting up client nodes... 2022-03-29T17:49:34.828 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-03-29T17:49:34.878 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-03-29T17:49:34.891 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-03-29T17:49:34.892 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2022-03-29T17:49:34.892 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-03-29T17:49:34.918 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2022-03-29T17:49:34.919 INFO:tasks.ceph:role: osd.0 2022-03-29T17:49:34.919 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com 2022-03-29T17:49:34.920 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-03-29T17:49:34.992 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-03-29T17:49:34.992 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2022-03-29T17:49:34.992 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-03-29T17:49:34.993 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-03-29T17:49:34.993 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-03-29T17:49:34.993 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2022-03-29T17:49:34.993 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-03-29T17:49:34.993 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-03-29T17:49:34.994 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-03-29T17:49:34.994 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-03-29T17:49:35.421 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2022-03-29T17:49:35.423 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2022-03-29T17:49:35.424 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-03-29T17:49:35.551 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-03-29T17:49:35.587 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-03-29T17:49:35.654 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2022-03-29T17:49:35.655 INFO:tasks.ceph:role: osd.1 2022-03-29T17:49:35.655 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com 2022-03-29T17:49:35.655 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-03-29T17:49:35.728 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-03-29T17:49:35.729 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2022-03-29T17:49:35.729 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-03-29T17:49:35.729 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-03-29T17:49:35.729 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-03-29T17:49:35.730 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2022-03-29T17:49:35.730 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-03-29T17:49:35.730 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-03-29T17:49:35.730 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-03-29T17:49:35.731 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-03-29T17:49:36.152 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2022-03-29T17:49:36.154 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2022-03-29T17:49:36.155 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-03-29T17:49:36.242 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-03-29T17:49:36.274 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-03-29T17:49:36.341 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2022-03-29T17:49:36.342 INFO:tasks.ceph:role: osd.2 2022-03-29T17:49:36.342 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com 2022-03-29T17:49:36.342 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-03-29T17:49:36.416 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-03-29T17:49:36.416 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2022-03-29T17:49:36.417 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-03-29T17:49:36.417 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-03-29T17:49:36.417 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-03-29T17:49:36.417 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2022-03-29T17:49:36.418 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-03-29T17:49:36.418 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-03-29T17:49:36.418 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-03-29T17:49:36.418 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-03-29T17:49:36.871 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2022-03-29T17:49:36.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2022-03-29T17:49:36.874 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-03-29T17:49:36.987 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-03-29T17:49:37.020 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-03-29T17:49:37.841 INFO:teuthology.orchestra.run.smithi138.stdout:warn: added seastar option --smp 1 2022-03-29T17:49:37.855 INFO:teuthology.orchestra.run.smithi138.stderr:==32348==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2022-03-29T17:49:37.860 INFO:teuthology.orchestra.run.smithi138.stderr:==32348==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc1577b000; bottom 0x7fb6fd37b000; size: 0x004518400000 (296759590912) 2022-03-29T17:49:37.861 INFO:teuthology.orchestra.run.smithi138.stderr:False positive error reports may follow 2022-03-29T17:49:37.861 INFO:teuthology.orchestra.run.smithi138.stderr:For details see https://github.com/google/sanitizers/issues/189 2022-03-29T17:49:37.909 INFO:teuthology.orchestra.run.smithi138.stdout:WARNING: debug mode. Not for benchmarking or production 2022-03-29T17:49:37.957 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,907 [shard 0] osd - open_all_classes 2022-03-29T17:49:37.958 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,907 [shard 0] osd - open_all_classes found journal 2022-03-29T17:49:37.958 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,907 [shard 0] osd - _get_class adding new class name journal 0x612000026000 2022-03-29T17:49:37.958 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,907 [shard 0] osd - _load_class journal from /usr/lib64/rados-classes/libcls_journal.so 2022-03-29T17:49:37.958 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_class journal status 3 2022-03-29T17:49:37.959 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.create flags 3 0x7fb6fbeaa7de 2022-03-29T17:49:37.959 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.get_order flags 1 0x7fb6fbeaa1de 2022-03-29T17:49:37.959 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.get_splay_width flags 1 0x7fb6fbeaa246 2022-03-29T17:49:37.960 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.get_pool_id flags 1 0x7fb6fbeaac38 2022-03-29T17:49:37.960 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.get_minimum_set flags 1 0x7fb6fbeaae60 2022-03-29T17:49:37.960 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.set_minimum_set flags 3 0x7fb6fbeaafb6 2022-03-29T17:49:37.960 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.get_active_set flags 1 0x7fb6fbeaaed2 2022-03-29T17:49:37.961 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.set_active_set flags 3 0x7fb6fbeab155 2022-03-29T17:49:37.961 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.get_client flags 1 0x7fb6fbeac4e4 2022-03-29T17:49:37.961 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.client_register flags 3 0x7fb6fbeadc5e 2022-03-29T17:49:37.961 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.client_update_data flags 3 0x7fb6fbeac7bf 2022-03-29T17:49:37.962 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.client_update_state flags 3 0x7fb6fbeacc9f 2022-03-29T17:49:37.962 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.client_unregister flags 3 0x7fb6fbeafe8a 2022-03-29T17:49:37.962 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.client_commit flags 3 0x7fb6fbeb0832 2022-03-29T17:49:37.962 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.client_list flags 1 0x7fb6fbeae30f 2022-03-29T17:49:37.962 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.get_next_tag_tid flags 1 0x7fb6fbeaaf44 2022-03-29T17:49:37.963 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.get_tag flags 1 0x7fb6fbeabf5d 2022-03-29T17:49:37.963 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.tag_create flags 3 0x7fb6fbeb0129 2022-03-29T17:49:37.963 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.tag_list flags 1 0x7fb6fbeae573 2022-03-29T17:49:37.963 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.guard_append flags 3 0x7fb6fbea9b5c 2022-03-29T17:49:37.964 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - register_cxx_method journal.append flags 3 0x7fb6fbeab2f4 2022-03-29T17:49:37.964 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - _load_class journal success 2022-03-29T17:49:37.964 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - open_all_classes found rgw 2022-03-29T17:49:37.964 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - _get_class adding new class name rgw 0x612000026180 2022-03-29T17:49:37.964 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,908 [shard 0] osd - _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so 2022-03-29T17:49:37.965 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,910 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw/cls_rgw.cc:4353: Loaded rgw class! 2022-03-29T17:49:37.965 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_class rgw status 3 2022-03-29T17:49:37.965 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_init_index flags 3 0x7fb6fbb5f175 2022-03-29T17:49:37.965 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7fb6fbb6eca1 2022-03-29T17:49:37.966 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_list flags 1 0x7fb6fbb7b331 2022-03-29T17:49:37.966 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_check_index flags 1 0x7fb6fbb79ea4 2022-03-29T17:49:37.966 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_rebuild_index flags 3 0x7fb6fbb7a70f 2022-03-29T17:49:37.966 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_update_stats flags 3 0x7fb6fbb705bd 2022-03-29T17:49:37.966 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_prepare_op flags 3 0x7fb6fbb6a3dd 2022-03-29T17:49:37.967 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_complete_op flags 3 0x7fb6fbb7225b 2022-03-29T17:49:37.967 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_link_olh flags 3 0x7fb6fbb7d49b 2022-03-29T17:49:37.967 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_unlink_instance flags 3 0x7fb6fbb74434 2022-03-29T17:49:37.967 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_read_olh_log flags 1 0x7fb6fbb75d50 2022-03-29T17:49:37.968 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7fb6fbb73dcb 2022-03-29T17:49:37.968 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bucket_clear_olh flags 3 0x7fb6fbb7361e 2022-03-29T17:49:37.968 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.obj_remove flags 3 0x7fb6fbb7748a 2022-03-29T17:49:37.968 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.obj_store_pg_ver flags 2 0x7fb6fbb63019 2022-03-29T17:49:37.968 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7fb6fbb62cda 2022-03-29T17:49:37.969 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.obj_check_mtime flags 1 0x7fb6fbb62a18 2022-03-29T17:49:37.969 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bi_get flags 1 0x7fb6fbb624fe 2022-03-29T17:49:37.969 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bi_put flags 3 0x7fb6fbb62304 2022-03-29T17:49:37.969 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bi_list flags 1 0x7fb6fbb78b1e 2022-03-29T17:49:37.969 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bi_log_list flags 1 0x7fb6fbb69f1a 2022-03-29T17:49:37.970 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bi_log_trim flags 3 0x7fb6fbb61d56 2022-03-29T17:49:37.970 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.dir_suggest_changes flags 3 0x7fb6fbb7093a 2022-03-29T17:49:37.970 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bi_log_resync flags 3 0x7fb6fbb6ffe3 2022-03-29T17:49:37.970 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.bi_log_stop flags 3 0x7fb6fbb6fa09 2022-03-29T17:49:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.user_usage_log_add flags 3 0x7fb6fbb6be48 2022-03-29T17:49:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.user_usage_log_read flags 1 0x7fb6fbb6b6db 2022-03-29T17:49:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.user_usage_log_trim flags 3 0x7fb6fbb6bbf9 2022-03-29T17:49:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.usage_log_clear flags 2 0x7fb6fbb5cdda 2022-03-29T17:49:37.972 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.gc_set_entry flags 3 0x7fb6fbb6538e 2022-03-29T17:49:37.972 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.gc_defer_entry flags 3 0x7fb6fbb65244 2022-03-29T17:49:37.972 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.gc_list flags 1 0x7fb6fbb639c9 2022-03-29T17:49:37.972 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.gc_remove flags 3 0x7fb6fbb7aeb9 2022-03-29T17:49:37.972 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.lc_get_entry flags 1 0x7fb6fbb64423 2022-03-29T17:49:37.973 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.lc_set_entry flags 3 0x7fb6fbb66271 2022-03-29T17:49:37.973 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.lc_rm_entry flags 3 0x7fb6fbb66122 2022-03-29T17:49:37.974 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.lc_get_next_entry flags 1 0x7fb6fbb61850 2022-03-29T17:49:37.974 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.lc_put_head flags 3 0x7fb6fbb615fb 2022-03-29T17:49:37.974 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.lc_get_head flags 1 0x7fb6fbb612c1 2022-03-29T17:49:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.lc_list_entries flags 1 0x7fb6fbb683ee 2022-03-29T17:49:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.reshard_add flags 3 0x7fb6fbb60cbc 2022-03-29T17:49:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.reshard_list flags 1 0x7fb6fbb65941 2022-03-29T17:49:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.reshard_get flags 1 0x7fb6fbb605af 2022-03-29T17:49:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.reshard_remove flags 3 0x7fb6fbb60188 2022-03-29T17:49:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.set_bucket_resharding flags 3 0x7fb6fbb6f74a 2022-03-29T17:49:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.clear_bucket_resharding flags 3 0x7fb6fbb6f4ee 2022-03-29T17:49:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.guard_bucket_resharding flags 1 0x7fb6fbb6f291 2022-03-29T17:49:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method rgw.get_bucket_resharding flags 1 0x7fb6fbb6eeea 2022-03-29T17:49:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - _load_class rgw success 2022-03-29T17:49:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - open_all_classes found cephfs 2022-03-29T17:49:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - _get_class adding new class name cephfs 0x612000026480 2022-03-29T17:49:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - _load_class cephfs from /usr/lib64/rados-classes/libcls_cephfs.so 2022-03-29T17:49:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:37,910 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cephfs/cls_cephfs.cc:201: loading cephfs 2022-03-29T17:49:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_class cephfs status 3 2022-03-29T17:49:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - register_cxx_method cephfs.accumulate_inode_metadata flags 3 0x7fb6fb8aa2f9 2022-03-29T17:49:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - _load_class cephfs success 2022-03-29T17:49:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - open_all_classes found numops 2022-03-29T17:49:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - _get_class adding new class name numops 0x612000026780 2022-03-29T17:49:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,910 [shard 0] osd - _load_class numops from /usr/lib64/rados-classes/libcls_numops.so 2022-03-29T17:49:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_class numops status 3 2022-03-29T17:49:37.980 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method numops.add flags 3 0x7fb6fb69c65f 2022-03-29T17:49:37.980 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method numops.mul flags 3 0x7fb6fb69bbe6 2022-03-29T17:49:37.980 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - _load_class numops success 2022-03-29T17:49:37.980 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - open_all_classes found lua 2022-03-29T17:49:37.980 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:37,911 [shard 0] osd - _get_class not permitted to load lua 2022-03-29T17:49:37.981 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - open_all_classes found test_remote_reads 2022-03-29T17:49:37.981 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:37,911 [shard 0] osd - _get_class not permitted to load test_remote_reads 2022-03-29T17:49:37.981 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - open_all_classes found rbd 2022-03-29T17:49:37.981 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - _get_class adding new class name rbd 0x612000026a80 2022-03-29T17:49:37.982 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so 2022-03-29T17:49:37.982 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_class rbd status 3 2022-03-29T17:49:37.982 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.create flags 3 0x7fb6fb43cec0 2022-03-29T17:49:37.982 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_features flags 1 0x7fb6fb42a5af 2022-03-29T17:49:37.983 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_features flags 3 0x7fb6fb42a7d2 2022-03-29T17:49:37.983 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_size flags 1 0x7fb6fb44fcb9 2022-03-29T17:49:37.983 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_size flags 3 0x7fb6fb42f75d 2022-03-29T17:49:37.983 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_snapcontext flags 1 0x7fb6fb42b27a 2022-03-29T17:49:37.984 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_object_prefix flags 1 0x7fb6fb429aaf 2022-03-29T17:49:37.984 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_data_pool flags 1 0x7fb6fb4295be 2022-03-29T17:49:37.984 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_snapshot_name flags 1 0x7fb6fb448f69 2022-03-29T17:49:37.984 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_snapshot_timestamp flags 1 0x7fb6fb448bc7 2022-03-29T17:49:37.984 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.snapshot_get flags 1 0x7fb6fb44ddcd 2022-03-29T17:49:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.snapshot_add flags 3 0x7fb6fb4549a5 2022-03-29T17:49:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.snapshot_remove flags 3 0x7fb6fb44d0e4 2022-03-29T17:49:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.snapshot_rename flags 3 0x7fb6fb44a938 2022-03-29T17:49:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.snapshot_trash_add flags 3 0x7fb6fb44982f 2022-03-29T17:49:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_all_features flags 1 0x7fb6fb4200bc 2022-03-29T17:49:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_parent flags 1 0x7fb6fb447d9f 2022-03-29T17:49:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_parent flags 3 0x7fb6fb43a007 2022-03-29T17:49:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.remove_parent flags 3 0x7fb6fb44ca52 2022-03-29T17:49:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.parent_get flags 1 0x7fb6fb44891f 2022-03-29T17:49:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.parent_overlap_get flags 1 0x7fb6fb450270 2022-03-29T17:49:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.parent_attach flags 3 0x7fb6fb430489 2022-03-29T17:49:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.parent_detach flags 3 0x7fb6fb44ca72 2022-03-29T17:49:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_protection_status flags 3 0x7fb6fb44f7f9 2022-03-29T17:49:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_protection_status flags 1 0x7fb6fb44e655 2022-03-29T17:49:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_stripe_unit_count flags 1 0x7fb6fb42d140 2022-03-29T17:49:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_stripe_unit_count flags 3 0x7fb6fb42c5c9 2022-03-29T17:49:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_create_timestamp flags 1 0x7fb6fb429e65 2022-03-29T17:49:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_access_timestamp flags 1 0x7fb6fb42a0d3 2022-03-29T17:49:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_modify_timestamp flags 1 0x7fb6fb42a341 2022-03-29T17:49:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_flags flags 1 0x7fb6fb450842 2022-03-29T17:49:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_flags flags 3 0x7fb6fb44ca92 2022-03-29T17:49:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.op_features_get flags 1 0x7fb6fb42944f 2022-03-29T17:49:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.op_features_set flags 3 0x7fb6fb42da28 2022-03-29T17:49:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.metadata_list flags 1 0x7fb6fb43ed62 2022-03-29T17:49:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.metadata_set flags 3 0x7fb6fb4475a7 2022-03-29T17:49:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.metadata_remove flags 3 0x7fb6fb426e87 2022-03-29T17:49:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.metadata_get flags 1 0x7fb6fb427065 2022-03-29T17:49:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.snapshot_get_limit flags 1 0x7fb6fb429731 2022-03-29T17:49:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.snapshot_set_limit flags 3 0x7fb6fb42c079 2022-03-29T17:49:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.child_attach flags 3 0x7fb6fb44d73a 2022-03-29T17:49:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.child_detach flags 3 0x7fb6fb44f0fc 2022-03-29T17:49:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.children_list flags 1 0x7fb6fb449329 2022-03-29T17:49:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.migration_set flags 3 0x7fb6fb42eb72 2022-03-29T17:49:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.migration_set_state flags 3 0x7fb6fb42edc9 2022-03-29T17:49:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.migration_get flags 1 0x7fb6fb42b075 2022-03-29T17:49:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.migration_remove flags 3 0x7fb6fb42d125 2022-03-29T17:49:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_modify_timestamp flags 3 0x7fb6fb4299aa 2022-03-29T17:49:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_access_timestamp flags 3 0x7fb6fb4298a5 2022-03-29T17:49:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.add_child flags 3 0x7fb6fb451af5 2022-03-29T17:49:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.remove_child flags 3 0x7fb6fb452052 2022-03-29T17:49:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_children flags 1 0x7fb6fb452633 2022-03-29T17:49:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.get_id flags 1 0x7fb6fb42248d 2022-03-29T17:49:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.set_id flags 3 0x7fb6fb4226f0 2022-03-29T17:49:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.dir_get_id flags 1 0x7fb6fb425c9d 2022-03-29T17:49:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.dir_get_name flags 1 0x7fb6fb42659c 2022-03-29T17:49:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.dir_list flags 1 0x7fb6fb43c5f8 2022-03-29T17:49:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.dir_add_image flags 3 0x7fb6fb440a41 2022-03-29T17:49:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,911 [shard 0] osd - register_cxx_method rbd.dir_remove_image flags 3 0x7fb6fb42641b 2022-03-29T17:49:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.dir_rename_image flags 3 0x7fb6fb44080d 2022-03-29T17:49:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.dir_state_assert flags 1 0x7fb6fb42933d 2022-03-29T17:49:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.dir_state_set flags 3 0x7fb6fb42bdf0 2022-03-29T17:49:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.object_map_load flags 1 0x7fb6fb440dee 2022-03-29T17:49:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.object_map_save flags 3 0x7fb6fb4412fc 2022-03-29T17:49:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.object_map_resize flags 3 0x7fb6fb442a9b 2022-03-29T17:49:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.object_map_update flags 3 0x7fb6fb441bce 2022-03-29T17:49:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.object_map_snap_add flags 3 0x7fb6fb440ec6 2022-03-29T17:49:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.object_map_snap_remove flags 3 0x7fb6fb4414bf 2022-03-29T17:49:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.snap_list flags 1 0x7fb6fb428f91 2022-03-29T17:49:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.snap_add flags 3 0x7fb6fb42b68e 2022-03-29T17:49:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.snap_remove flags 3 0x7fb6fb422b41 2022-03-29T17:49:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.snap_rename flags 3 0x7fb6fb4232a2 2022-03-29T17:49:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_uuid_get flags 1 0x7fb6fb428cef 2022-03-29T17:49:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_uuid_set flags 3 0x7fb6fb435253 2022-03-29T17:49:38.000 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_mode_get flags 1 0x7fb6fb423c4b 2022-03-29T17:49:38.000 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_mode_set flags 3 0x7fb6fb438ba6 2022-03-29T17:49:38.000 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_peer_ping flags 3 0x7fb6fb4473fb 2022-03-29T17:49:38.000 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_peer_list flags 1 0x7fb6fb439035 2022-03-29T17:49:38.000 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_peer_add flags 3 0x7fb6fb43a2de 2022-03-29T17:49:38.001 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_peer_remove flags 3 0x7fb6fb43113d 2022-03-29T17:49:38.001 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_client flags 3 0x7fb6fb430b04 2022-03-29T17:49:38.001 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_cluster flags 3 0x7fb6fb43918f 2022-03-29T17:49:38.001 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_direction flags 3 0x7fb6fb430ddb 2022-03-29T17:49:38.002 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_list flags 1 0x7fb6fb43bcae 2022-03-29T17:49:38.002 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_get_image_id flags 1 0x7fb6fb431e8c 2022-03-29T17:49:38.002 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_get flags 1 0x7fb6fb43149a 2022-03-29T17:49:38.003 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_set flags 3 0x7fb6fb431d0f 2022-03-29T17:49:38.003 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_remove flags 3 0x7fb6fb437f25 2022-03-29T17:49:38.003 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_status_set flags 11 0x7fb6fb4377da 2022-03-29T17:49:38.004 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove flags 3 0x7fb6fb43802f 2022-03-29T17:49:38.004 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get flags 1 0x7fb6fb439e5f 2022-03-29T17:49:38.005 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_status_list flags 1 0x7fb6fb443d7f 2022-03-29T17:49:38.005 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get_summary flags 1 0x7fb6fb444570 2022-03-29T17:49:38.006 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove_down flags 3 0x7fb6fb436f42 2022-03-29T17:49:38.006 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_get flags 1 0x7fb6fb4372cd 2022-03-29T17:49:38.007 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_list flags 1 0x7fb6fb44504d 2022-03-29T17:49:38.007 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_instances_list flags 1 0x7fb6fb435851 2022-03-29T17:49:38.007 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_instances_add flags 11 0x7fb6fb4324de 2022-03-29T17:49:38.008 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_instances_remove flags 3 0x7fb6fb4326ef 2022-03-29T17:49:38.008 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_map_list flags 1 0x7fb6fb43505e 2022-03-29T17:49:38.008 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_map_update flags 2 0x7fb6fb432841 2022-03-29T17:49:38.009 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_map_remove flags 2 0x7fb6fb432bbd 2022-03-29T17:49:38.009 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_unlink_peer flags 3 0x7fb6fb453b3c 2022-03-29T17:49:38.010 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_set_copy_progress flags 3 0x7fb6fb452e49 2022-03-29T17:49:38.010 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_dir_list flags 1 0x7fb6fb43b196 2022-03-29T17:49:38.010 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_dir_add flags 3 0x7fb6fb43fca7 2022-03-29T17:49:38.010 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_dir_remove flags 3 0x7fb6fb426cc2 2022-03-29T17:49:38.011 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_dir_rename flags 3 0x7fb6fb43fec2 2022-03-29T17:49:38.011 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_image_remove flags 3 0x7fb6fb42032a 2022-03-29T17:49:38.011 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_image_list flags 1 0x7fb6fb435c9a 2022-03-29T17:49:38.011 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_image_set flags 3 0x7fb6fb423cb3 2022-03-29T17:49:38.012 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.image_group_add flags 3 0x7fb6fb42db03 2022-03-29T17:49:38.012 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.image_group_remove flags 3 0x7fb6fb42df46 2022-03-29T17:49:38.012 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.image_group_get flags 1 0x7fb6fb429c65 2022-03-29T17:49:38.012 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_snap_set flags 3 0x7fb6fb445ae6 2022-03-29T17:49:38.013 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_snap_remove flags 3 0x7fb6fb427b76 2022-03-29T17:49:38.013 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_snap_get_by_id flags 1 0x7fb6fb427d09 2022-03-29T17:49:38.013 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.group_snap_list flags 1 0x7fb6fb43ac44 2022-03-29T17:49:38.013 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.trash_add flags 3 0x7fb6fb43e79b 2022-03-29T17:49:38.013 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.trash_remove flags 3 0x7fb6fb432de3 2022-03-29T17:49:38.014 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.trash_list flags 1 0x7fb6fb445ece 2022-03-29T17:49:38.014 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.trash_get flags 1 0x7fb6fb4330e8 2022-03-29T17:49:38.014 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.trash_state_set flags 3 0x7fb6fb43336e 2022-03-29T17:49:38.014 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.namespace_add flags 3 0x7fb6fb4337a6 2022-03-29T17:49:38.015 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.namespace_remove flags 3 0x7fb6fb4339d1 2022-03-29T17:49:38.015 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.namespace_list flags 1 0x7fb6fb433bba 2022-03-29T17:49:38.015 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.copyup flags 3 0x7fb6fb42016b 2022-03-29T17:49:38.015 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.sparse_copyup flags 3 0x7fb6fb450e66 2022-03-29T17:49:38.016 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.assert_snapc_seq flags 3 0x7fb6fb42021b 2022-03-29T17:49:38.016 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method rbd.sparsify flags 3 0x7fb6fb424074 2022-03-29T17:49:38.016 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - _load_class rbd success 2022-03-29T17:49:38.016 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - open_all_classes found queue 2022-03-29T17:49:38.017 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - _get_class adding new class name queue 0x612000026c00 2022-03-29T17:49:38.017 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - _load_class queue from /usr/lib64/rados-classes/libcls_queue.so 2022-03-29T17:49:38.017 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,912 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/queue/cls_queue.cc:125: Loaded queue class! 2022-03-29T17:49:38.017 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_class queue status 3 2022-03-29T17:49:38.017 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method queue.queue_init flags 3 0x7fb6fb1e4256 2022-03-29T17:49:38.018 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method queue.queue_get_capacity flags 1 0x7fb6fb1e39e9 2022-03-29T17:49:38.018 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,912 [shard 0] osd - register_cxx_method queue.queue_enqueue flags 3 0x7fb6fb1e452e 2022-03-29T17:49:38.018 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method queue.queue_list_entries flags 1 0x7fb6fb1e3daf 2022-03-29T17:49:38.018 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method queue.queue_remove_entries flags 3 0x7fb6fb1e3b81 2022-03-29T17:49:38.019 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _load_class queue success 2022-03-29T17:49:38.019 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - open_all_classes found user 2022-03-29T17:49:38.019 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _get_class adding new class name user 0x612000026f00 2022-03-29T17:49:38.019 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _load_class user from /usr/lib64/rados-classes/libcls_user.so 2022-03-29T17:49:38.019 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,913 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/user/cls_user.cc:506: Loaded user class! 2022-03-29T17:49:38.020 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_class user status 3 2022-03-29T17:49:38.020 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method user.set_buckets_info flags 3 0x7fb6fafcac4b 2022-03-29T17:49:38.020 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method user.complete_stats_sync flags 3 0x7fb6fafca9a3 2022-03-29T17:49:38.021 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method user.remove_bucket flags 3 0x7fb6fafca1be 2022-03-29T17:49:38.021 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method user.list_buckets flags 1 0x7fb6fafcb733 2022-03-29T17:49:38.021 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method user.get_header flags 1 0x7fb6fafc9fc7 2022-03-29T17:49:38.022 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method user.reset_user_stats flags 3 0x7fb6fafc954c 2022-03-29T17:49:38.022 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method user.reset_user_stats2 flags 3 0x7fb6fafcc19b 2022-03-29T17:49:38.022 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _load_class user success 2022-03-29T17:49:38.022 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - open_all_classes found cas 2022-03-29T17:49:38.022 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _get_class adding new class name cas 0x612000027380 2022-03-29T17:49:38.023 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _load_class cas from /usr/lib64/rados-classes/libcls_cas.so 2022-03-29T17:49:38.023 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,913 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cas/cls_cas.cc:211: Loaded cas class! 2022-03-29T17:49:38.023 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_class cas status 3 2022-03-29T17:49:38.024 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method cas.chunk_create_or_get_ref flags 3 0x7fb6fadb3b38 2022-03-29T17:49:38.024 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method cas.chunk_get_ref flags 3 0x7fb6fadb3848 2022-03-29T17:49:38.024 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method cas.chunk_put_ref flags 3 0x7fb6fadb34c4 2022-03-29T17:49:38.024 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method cas.references_chunk flags 1 0x7fb6fadb31fc 2022-03-29T17:49:38.024 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _load_class cas success 2022-03-29T17:49:38.025 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - open_all_classes found fifo 2022-03-29T17:49:38.025 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _get_class adding new class name fifo 0x612000027680 2022-03-29T17:49:38.025 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _load_class fifo from /usr/lib64/rados-classes/libcls_fifo.so 2022-03-29T17:49:38.025 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_class fifo status 3 2022-03-29T17:49:38.026 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method fifo.create_meta flags 3 0x7fb6fab74dee 2022-03-29T17:49:38.026 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method fifo.get_meta flags 1 0x7fb6fab760b2 2022-03-29T17:49:38.026 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method fifo.update_meta flags 3 0x7fb6fab7657b 2022-03-29T17:49:38.026 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method fifo.init_part flags 3 0x7fb6fab73cf2 2022-03-29T17:49:38.026 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method fifo.push_part flags 3 0x7fb6fab743e5 2022-03-29T17:49:38.027 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method fifo.trim_part flags 3 0x7fb6fab73782 2022-03-29T17:49:38.027 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method fifo.part_list flags 1 0x7fb6fab72ed9 2022-03-29T17:49:38.028 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - register_cxx_method fifo.get_part_info flags 1 0x7fb6fab7277e 2022-03-29T17:49:38.028 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _load_class fifo success 2022-03-29T17:49:38.028 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - open_all_classes found cmpomap 2022-03-29T17:49:38.029 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _get_class adding new class name cmpomap 0x612000027980 2022-03-29T17:49:38.029 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,913 [shard 0] osd - _load_class cmpomap from /usr/lib64/rados-classes/libcls_cmpomap.so 2022-03-29T17:49:38.030 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,914 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cmpomap/server.cc:287: Loaded cmpomap class! 2022-03-29T17:49:38.030 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_class cmpomap status 3 2022-03-29T17:49:38.030 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method cmpomap.cmp_vals flags 1 0x7fb6fa9572e7 2022-03-29T17:49:38.030 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method cmpomap.cmp_set_vals flags 3 0x7fb6fa95670e 2022-03-29T17:49:38.031 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method cmpomap.cmp_rm_keys flags 3 0x7fb6fa956136 2022-03-29T17:49:38.031 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _load_class cmpomap success 2022-03-29T17:49:38.031 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - open_all_classes found hello 2022-03-29T17:49:38.031 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _get_class adding new class name hello 0x612000027e00 2022-03-29T17:49:38.031 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _load_class hello from /usr/lib64/rados-classes/libcls_hello.so 2022-03-29T17:49:38.032 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:37,914 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/hello/cls_hello.cc:316: loading cls_hello 2022-03-29T17:49:38.032 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_class hello status 3 2022-03-29T17:49:38.032 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.say_hello flags 1 0x7fb6fa746bf6 2022-03-29T17:49:38.032 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.record_hello flags 10 0x7fb6fa746ea9 2022-03-29T17:49:38.033 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.write_return_data flags 2 0x7fb6fa746d7f 2022-03-29T17:49:38.033 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.writes_dont_return_data flags 2 0x7fb6fa746d7f 2022-03-29T17:49:38.034 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.write_too_much_return_data flags 2 0x7fb6fa746c6d 2022-03-29T17:49:38.034 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.replay flags 1 0x7fb6fa746bce 2022-03-29T17:49:38.034 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.turn_it_to_11 flags 11 0x7fb6fa747808 2022-03-29T17:49:38.035 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.bad_reader flags 2 0x7fb6fa746baf 2022-03-29T17:49:38.035 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method hello.bad_writer flags 1 0x7fb6fa746b9d 2022-03-29T17:49:38.036 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _load_class hello success 2022-03-29T17:49:38.036 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - open_all_classes found log 2022-03-29T17:49:38.037 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _get_class adding new class name log 0x612000028100 2022-03-29T17:49:38.037 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _load_class log from /usr/lib64/rados-classes/libcls_log.so 2022-03-29T17:49:38.037 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,914 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/log/cls_log.cc:305: Loaded log class! 2022-03-29T17:49:38.037 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_class log status 3 2022-03-29T17:49:38.037 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method log.add flags 3 0x7fb6fa535f1f 2022-03-29T17:49:38.038 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method log.list flags 1 0x7fb6fa536a5d 2022-03-29T17:49:38.038 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method log.trim flags 3 0x7fb6fa5362cf 2022-03-29T17:49:38.039 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_cxx_method log.info flags 1 0x7fb6fa535af7 2022-03-29T17:49:38.039 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _load_class log success 2022-03-29T17:49:38.040 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - open_all_classes found otp 2022-03-29T17:49:38.040 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _get_class adding new class name otp 0x612000028580 2022-03-29T17:49:38.040 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - _load_class otp from /usr/lib64/rados-classes/libcls_otp.so 2022-03-29T17:49:38.041 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,914 [shard 0] osd - register_class otp status 3 2022-03-29T17:49:38.041 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method otp.otp_set flags 3 0x7fb6fa32237f 2022-03-29T17:49:38.042 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method otp.otp_get flags 1 0x7fb6fa32198f 2022-03-29T17:49:38.042 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method otp.otp_check flags 3 0x7fb6fa320fc3 2022-03-29T17:49:38.043 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method otp.otp_get_result flags 1 0x7fb6fa320902 2022-03-29T17:49:38.043 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method otp.otp_remove flags 3 0x7fb6fa321700 2022-03-29T17:49:38.043 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method otp.get_current_time flags 1 0x7fb6fa3205f0 2022-03-29T17:49:38.043 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _load_class otp success 2022-03-29T17:49:38.044 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - open_all_classes found sdk 2022-03-29T17:49:38.044 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:37,915 [shard 0] osd - _get_class not permitted to load sdk 2022-03-29T17:49:38.044 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - open_all_classes found timeindex 2022-03-29T17:49:38.044 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _get_class adding new class name timeindex 0x612000028700 2022-03-29T17:49:38.045 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _load_class timeindex from /usr/lib64/rados-classes/libcls_timeindex.so 2022-03-29T17:49:38.045 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,915 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/timeindex/cls_timeindex.cc:247: Loaded timeindex class! 2022-03-29T17:49:38.045 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_class timeindex status 3 2022-03-29T17:49:38.045 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method timeindex.add flags 3 0x7fb6f9ef3894 2022-03-29T17:49:38.045 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method timeindex.list flags 1 0x7fb6f9ef3c23 2022-03-29T17:49:38.046 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method timeindex.trim flags 3 0x7fb6f9ef343d 2022-03-29T17:49:38.046 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _load_class timeindex success 2022-03-29T17:49:38.046 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - open_all_classes found lock 2022-03-29T17:49:38.047 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _get_class adding new class name lock 0x612000028b80 2022-03-29T17:49:38.047 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _load_class lock from /usr/lib64/rados-classes/libcls_lock.so 2022-03-29T17:49:38.047 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_class lock status 3 2022-03-29T17:49:38.047 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method lock.lock flags 11 0x7fb6f9ce04af 2022-03-29T17:49:38.047 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method lock.unlock flags 11 0x7fb6f9cdde27 2022-03-29T17:49:38.048 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method lock.break_lock flags 3 0x7fb6f9cddccd 2022-03-29T17:49:38.048 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method lock.get_info flags 1 0x7fb6f9cdea33 2022-03-29T17:49:38.049 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method lock.list_locks flags 1 0x7fb6f9cdc26b 2022-03-29T17:49:38.049 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method lock.assert_locked flags 9 0x7fb6f9cdd3bd 2022-03-29T17:49:38.049 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method lock.set_cookie flags 11 0x7fb6f9cddff4 2022-03-29T17:49:38.050 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _load_class lock success 2022-03-29T17:49:38.050 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - open_all_classes found 2pc_queue 2022-03-29T17:49:38.051 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _get_class adding new class name 2pc_queue 0x612000028d00 2022-03-29T17:49:38.051 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _load_class 2pc_queue from /usr/lib64/rados-classes/libcls_2pc_queue.so 2022-03-29T17:49:38.052 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,915 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/2pc_queue/cls_2pc_queue.cc:575: Loaded 2pc queue class! 2022-03-29T17:49:38.052 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_class 2pc_queue status 3 2022-03-29T17:49:38.052 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_init flags 3 0x7fb6f9a9f6a3 2022-03-29T17:49:38.053 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_get_capacity flags 1 0x7fb6f9a9ec19 2022-03-29T17:49:38.053 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_reserve flags 3 0x7fb6f9aa2504 2022-03-29T17:49:38.053 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_commit flags 3 0x7fb6f9a9fb0d 2022-03-29T17:49:38.053 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_abort flags 3 0x7fb6f9aa128f 2022-03-29T17:49:38.054 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_reservations flags 1 0x7fb6f9aa1c8c 2022-03-29T17:49:38.054 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_entries flags 1 0x7fb6f9a9f076 2022-03-29T17:49:38.054 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_remove_entries flags 3 0x7fb6f9a9ee39 2022-03-29T17:49:38.054 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_expire_reservations flags 3 0x7fb6f9aa07d0 2022-03-29T17:49:38.055 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _load_class 2pc_queue success 2022-03-29T17:49:38.055 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - open_all_classes found refcount 2022-03-29T17:49:38.055 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _get_class adding new class name refcount 0x612000029000 2022-03-29T17:49:38.055 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,915 [shard 0] osd - _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so 2022-03-29T17:49:38.056 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,917 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/refcount/cls_refcount.cc:199: Loaded refcount class! 2022-03-29T17:49:38.056 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_class refcount status 3 2022-03-29T17:49:38.056 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method refcount.get flags 3 0x7fb6f97d7c89 2022-03-29T17:49:38.056 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method refcount.put flags 3 0x7fb6f97d794c 2022-03-29T17:49:38.056 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method refcount.set flags 3 0x7fb6f97d70c1 2022-03-29T17:49:38.057 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method refcount.read flags 1 0x7fb6f97d7602 2022-03-29T17:49:38.057 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - _load_class refcount success 2022-03-29T17:49:38.057 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - open_all_classes found version 2022-03-29T17:49:38.057 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - _get_class adding new class name version 0x612000029300 2022-03-29T17:49:38.057 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - _load_class version from /usr/lib64/rados-classes/libcls_version.so 2022-03-29T17:49:38.058 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,917 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/version/cls_version.cc:218: Loaded version class! 2022-03-29T17:49:38.058 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_class version status 3 2022-03-29T17:49:38.058 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method version.set flags 3 0x7fb6f95263d7 2022-03-29T17:49:38.058 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method version.inc flags 3 0x7fb6f95266e1 2022-03-29T17:49:38.059 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method version.inc_conds flags 3 0x7fb6f95266e1 2022-03-29T17:49:38.059 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method version.read flags 1 0x7fb6f952621c 2022-03-29T17:49:38.059 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method version.check_conds flags 1 0x7fb6f95264f9 2022-03-29T17:49:38.059 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - _load_class version success 2022-03-29T17:49:38.060 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - open_all_classes found rgw_gc 2022-03-29T17:49:38.060 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - _get_class adding new class name rgw_gc 0x612000029780 2022-03-29T17:49:38.060 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - _load_class rgw_gc from /usr/lib64/rados-classes/libcls_rgw_gc.so 2022-03-29T17:49:38.060 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,917 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw_gc/cls_rgw_gc.cc:539: Loaded rgw gc class! 2022-03-29T17:49:38.061 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_class rgw_gc status 3 2022-03-29T17:49:38.061 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_init flags 3 0x7fb6f92e9a31 2022-03-29T17:49:38.061 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_enqueue flags 3 0x7fb6f92e9130 2022-03-29T17:49:38.061 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_list_entries flags 1 0x7fb6f92eb2eb 2022-03-29T17:49:38.061 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_remove_entries flags 3 0x7fb6f92e9d97 2022-03-29T17:49:38.062 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_update_entry flags 3 0x7fb6f92ece0e 2022-03-29T17:49:38.062 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:37,917 [shard 0] osd - _load_class rgw_gc success 2022-03-29T17:49:38.062 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:37,917 [shard 0] osd - OSD: nonce is 32348 2022-03-29T17:49:40.346 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:40,344 [shard 0] osd - _write_superblock writing superblock cluster_fsid 3be68405-f255-4011-9a3b-ba646329d493 osd_fsid 3e4be414-1d2c-4bc5-b6b2-d1f7948d012a 2022-03-29T17:49:40.346 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:40,344 [shard 0] osd - OSD::_write_superblock: do_transaction... 2022-03-29T17:49:40.357 INFO:teuthology.orchestra.run.smithi138.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 3be68405-f255-4011-9a3b-ba646329d493 2022-03-29T17:49:40.357 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:40,355 [shard 0] osd - stop 2022-03-29T17:49:40.358 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:40,355 [shard 0] osd - prepared to stop 2022-03-29T17:49:40.358 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:40,355 [shard 0] osd - stop 2022-03-29T17:49:40.358 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:40,355 [shard 0] alienstore - umount 2022-03-29T17:49:40.862 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:40,861 [shard 0] monc - stop 2022-03-29T17:49:40.863 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:40,861 [shard 0] mgrc - stop 2022-03-29T17:49:41.332 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-03-29T17:49:41.358 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-03-29T17:49:42.178 INFO:teuthology.orchestra.run.smithi138.stdout:warn: added seastar option --smp 1 2022-03-29T17:49:42.189 INFO:teuthology.orchestra.run.smithi138.stderr:==33163==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2022-03-29T17:49:42.194 INFO:teuthology.orchestra.run.smithi138.stderr:==33163==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe2d856000; bottom 0x7febca522000; size: 0x001263334000 (78973714432) 2022-03-29T17:49:42.195 INFO:teuthology.orchestra.run.smithi138.stderr:False positive error reports may follow 2022-03-29T17:49:42.195 INFO:teuthology.orchestra.run.smithi138.stderr:For details see https://github.com/google/sanitizers/issues/189 2022-03-29T17:49:42.226 INFO:teuthology.orchestra.run.smithi138.stdout:WARNING: debug mode. Not for benchmarking or production 2022-03-29T17:49:42.267 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,224 [shard 0] osd - open_all_classes 2022-03-29T17:49:42.267 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,224 [shard 0] osd - open_all_classes found journal 2022-03-29T17:49:42.268 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,224 [shard 0] osd - _get_class adding new class name journal 0x612000026000 2022-03-29T17:49:42.268 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,224 [shard 0] osd - _load_class journal from /usr/lib64/rados-classes/libcls_journal.so 2022-03-29T17:49:42.268 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_class journal status 3 2022-03-29T17:49:42.268 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.create flags 3 0x7febc90517de 2022-03-29T17:49:42.269 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.get_order flags 1 0x7febc90511de 2022-03-29T17:49:42.269 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.get_splay_width flags 1 0x7febc9051246 2022-03-29T17:49:42.269 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.get_pool_id flags 1 0x7febc9051c38 2022-03-29T17:49:42.269 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.get_minimum_set flags 1 0x7febc9051e60 2022-03-29T17:49:42.269 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.set_minimum_set flags 3 0x7febc9051fb6 2022-03-29T17:49:42.270 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.get_active_set flags 1 0x7febc9051ed2 2022-03-29T17:49:42.270 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.set_active_set flags 3 0x7febc9052155 2022-03-29T17:49:42.270 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.get_client flags 1 0x7febc90534e4 2022-03-29T17:49:42.271 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.client_register flags 3 0x7febc9054c5e 2022-03-29T17:49:42.271 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.client_update_data flags 3 0x7febc90537bf 2022-03-29T17:49:42.271 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.client_update_state flags 3 0x7febc9053c9f 2022-03-29T17:49:42.271 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.client_unregister flags 3 0x7febc9056e8a 2022-03-29T17:49:42.272 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.client_commit flags 3 0x7febc9057832 2022-03-29T17:49:42.272 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.client_list flags 1 0x7febc905530f 2022-03-29T17:49:42.272 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.get_next_tag_tid flags 1 0x7febc9051f44 2022-03-29T17:49:42.272 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.get_tag flags 1 0x7febc9052f5d 2022-03-29T17:49:42.272 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.tag_create flags 3 0x7febc9057129 2022-03-29T17:49:42.273 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.tag_list flags 1 0x7febc9055573 2022-03-29T17:49:42.273 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.guard_append flags 3 0x7febc9050b5c 2022-03-29T17:49:42.273 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - register_cxx_method journal.append flags 3 0x7febc90522f4 2022-03-29T17:49:42.273 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - _load_class journal success 2022-03-29T17:49:42.274 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - open_all_classes found rgw 2022-03-29T17:49:42.274 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - _get_class adding new class name rgw 0x612000026180 2022-03-29T17:49:42.274 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,225 [shard 0] osd - _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so 2022-03-29T17:49:42.274 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,227 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw/cls_rgw.cc:4353: Loaded rgw class! 2022-03-29T17:49:42.275 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_class rgw status 3 2022-03-29T17:49:42.275 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_init_index flags 3 0x7febc8d06175 2022-03-29T17:49:42.275 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7febc8d15ca1 2022-03-29T17:49:42.275 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_list flags 1 0x7febc8d22331 2022-03-29T17:49:42.275 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_check_index flags 1 0x7febc8d20ea4 2022-03-29T17:49:42.276 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_rebuild_index flags 3 0x7febc8d2170f 2022-03-29T17:49:42.276 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_update_stats flags 3 0x7febc8d175bd 2022-03-29T17:49:42.276 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_prepare_op flags 3 0x7febc8d113dd 2022-03-29T17:49:42.276 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_complete_op flags 3 0x7febc8d1925b 2022-03-29T17:49:42.277 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_link_olh flags 3 0x7febc8d2449b 2022-03-29T17:49:42.277 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_unlink_instance flags 3 0x7febc8d1b434 2022-03-29T17:49:42.277 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_read_olh_log flags 1 0x7febc8d1cd50 2022-03-29T17:49:42.277 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7febc8d1adcb 2022-03-29T17:49:42.277 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,227 [shard 0] osd - register_cxx_method rgw.bucket_clear_olh flags 3 0x7febc8d1a61e 2022-03-29T17:49:42.278 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.obj_remove flags 3 0x7febc8d1e48a 2022-03-29T17:49:42.278 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.obj_store_pg_ver flags 2 0x7febc8d0a019 2022-03-29T17:49:42.278 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7febc8d09cda 2022-03-29T17:49:42.278 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.obj_check_mtime flags 1 0x7febc8d09a18 2022-03-29T17:49:42.279 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.bi_get flags 1 0x7febc8d094fe 2022-03-29T17:49:42.279 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.bi_put flags 3 0x7febc8d09304 2022-03-29T17:49:42.279 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.bi_list flags 1 0x7febc8d1fb1e 2022-03-29T17:49:42.279 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.bi_log_list flags 1 0x7febc8d10f1a 2022-03-29T17:49:42.279 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.bi_log_trim flags 3 0x7febc8d08d56 2022-03-29T17:49:42.280 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.dir_suggest_changes flags 3 0x7febc8d1793a 2022-03-29T17:49:42.280 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.bi_log_resync flags 3 0x7febc8d16fe3 2022-03-29T17:49:42.280 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.bi_log_stop flags 3 0x7febc8d16a09 2022-03-29T17:49:42.280 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.user_usage_log_add flags 3 0x7febc8d12e48 2022-03-29T17:49:42.280 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.user_usage_log_read flags 1 0x7febc8d126db 2022-03-29T17:49:42.281 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.user_usage_log_trim flags 3 0x7febc8d12bf9 2022-03-29T17:49:42.281 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.usage_log_clear flags 2 0x7febc8d03dda 2022-03-29T17:49:42.281 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.gc_set_entry flags 3 0x7febc8d0c38e 2022-03-29T17:49:42.282 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.gc_defer_entry flags 3 0x7febc8d0c244 2022-03-29T17:49:42.282 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.gc_list flags 1 0x7febc8d0a9c9 2022-03-29T17:49:42.282 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.gc_remove flags 3 0x7febc8d21eb9 2022-03-29T17:49:42.282 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.lc_get_entry flags 1 0x7febc8d0b423 2022-03-29T17:49:42.282 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.lc_set_entry flags 3 0x7febc8d0d271 2022-03-29T17:49:42.283 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.lc_rm_entry flags 3 0x7febc8d0d122 2022-03-29T17:49:42.283 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.lc_get_next_entry flags 1 0x7febc8d08850 2022-03-29T17:49:42.283 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.lc_put_head flags 3 0x7febc8d085fb 2022-03-29T17:49:42.283 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.lc_get_head flags 1 0x7febc8d082c1 2022-03-29T17:49:42.284 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.lc_list_entries flags 1 0x7febc8d0f3ee 2022-03-29T17:49:42.284 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.reshard_add flags 3 0x7febc8d07cbc 2022-03-29T17:49:42.284 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.reshard_list flags 1 0x7febc8d0c941 2022-03-29T17:49:42.284 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.reshard_get flags 1 0x7febc8d075af 2022-03-29T17:49:42.284 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.reshard_remove flags 3 0x7febc8d07188 2022-03-29T17:49:42.285 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.set_bucket_resharding flags 3 0x7febc8d1674a 2022-03-29T17:49:42.285 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.clear_bucket_resharding flags 3 0x7febc8d164ee 2022-03-29T17:49:42.285 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.guard_bucket_resharding flags 1 0x7febc8d16291 2022-03-29T17:49:42.285 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method rgw.get_bucket_resharding flags 1 0x7febc8d15eea 2022-03-29T17:49:42.286 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _load_class rgw success 2022-03-29T17:49:42.286 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - open_all_classes found cephfs 2022-03-29T17:49:42.286 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _get_class adding new class name cephfs 0x612000026480 2022-03-29T17:49:42.286 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _load_class cephfs from /usr/lib64/rados-classes/libcls_cephfs.so 2022-03-29T17:49:42.286 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:42,228 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cephfs/cls_cephfs.cc:201: loading cephfs 2022-03-29T17:49:42.287 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_class cephfs status 3 2022-03-29T17:49:42.287 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method cephfs.accumulate_inode_metadata flags 3 0x7febc8a512f9 2022-03-29T17:49:42.287 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _load_class cephfs success 2022-03-29T17:49:42.287 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - open_all_classes found numops 2022-03-29T17:49:42.288 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _get_class adding new class name numops 0x612000026780 2022-03-29T17:49:42.288 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _load_class numops from /usr/lib64/rados-classes/libcls_numops.so 2022-03-29T17:49:42.288 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_class numops status 3 2022-03-29T17:49:42.288 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method numops.add flags 3 0x7febc884365f 2022-03-29T17:49:42.288 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - register_cxx_method numops.mul flags 3 0x7febc8842be6 2022-03-29T17:49:42.289 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _load_class numops success 2022-03-29T17:49:42.289 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - open_all_classes found lua 2022-03-29T17:49:42.289 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:42,228 [shard 0] osd - _get_class not permitted to load lua 2022-03-29T17:49:42.289 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - open_all_classes found test_remote_reads 2022-03-29T17:49:42.290 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:42,228 [shard 0] osd - _get_class not permitted to load test_remote_reads 2022-03-29T17:49:42.290 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - open_all_classes found rbd 2022-03-29T17:49:42.290 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _get_class adding new class name rbd 0x612000026a80 2022-03-29T17:49:42.290 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,228 [shard 0] osd - _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so 2022-03-29T17:49:42.290 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_class rbd status 3 2022-03-29T17:49:42.291 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.create flags 3 0x7febc85e3ec0 2022-03-29T17:49:42.291 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_features flags 1 0x7febc85d15af 2022-03-29T17:49:42.291 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_features flags 3 0x7febc85d17d2 2022-03-29T17:49:42.291 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_size flags 1 0x7febc85f6cb9 2022-03-29T17:49:42.292 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_size flags 3 0x7febc85d675d 2022-03-29T17:49:42.292 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_snapcontext flags 1 0x7febc85d227a 2022-03-29T17:49:42.292 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_object_prefix flags 1 0x7febc85d0aaf 2022-03-29T17:49:42.293 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_data_pool flags 1 0x7febc85d05be 2022-03-29T17:49:42.293 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_snapshot_name flags 1 0x7febc85eff69 2022-03-29T17:49:42.293 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_snapshot_timestamp flags 1 0x7febc85efbc7 2022-03-29T17:49:42.293 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snapshot_get flags 1 0x7febc85f4dcd 2022-03-29T17:49:42.293 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snapshot_add flags 3 0x7febc85fb9a5 2022-03-29T17:49:42.294 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snapshot_remove flags 3 0x7febc85f40e4 2022-03-29T17:49:42.294 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snapshot_rename flags 3 0x7febc85f1938 2022-03-29T17:49:42.294 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snapshot_trash_add flags 3 0x7febc85f082f 2022-03-29T17:49:42.294 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_all_features flags 1 0x7febc85c70bc 2022-03-29T17:49:42.295 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_parent flags 1 0x7febc85eed9f 2022-03-29T17:49:42.295 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_parent flags 3 0x7febc85e1007 2022-03-29T17:49:42.295 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.remove_parent flags 3 0x7febc85f3a52 2022-03-29T17:49:42.295 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.parent_get flags 1 0x7febc85ef91f 2022-03-29T17:49:42.295 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.parent_overlap_get flags 1 0x7febc85f7270 2022-03-29T17:49:42.296 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.parent_attach flags 3 0x7febc85d7489 2022-03-29T17:49:42.296 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.parent_detach flags 3 0x7febc85f3a72 2022-03-29T17:49:42.296 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_protection_status flags 3 0x7febc85f67f9 2022-03-29T17:49:42.296 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_protection_status flags 1 0x7febc85f5655 2022-03-29T17:49:42.296 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_stripe_unit_count flags 1 0x7febc85d4140 2022-03-29T17:49:42.297 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_stripe_unit_count flags 3 0x7febc85d35c9 2022-03-29T17:49:42.297 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_create_timestamp flags 1 0x7febc85d0e65 2022-03-29T17:49:42.297 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_access_timestamp flags 1 0x7febc85d10d3 2022-03-29T17:49:42.297 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_modify_timestamp flags 1 0x7febc85d1341 2022-03-29T17:49:42.298 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_flags flags 1 0x7febc85f7842 2022-03-29T17:49:42.298 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_flags flags 3 0x7febc85f3a92 2022-03-29T17:49:42.298 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.op_features_get flags 1 0x7febc85d044f 2022-03-29T17:49:42.298 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.op_features_set flags 3 0x7febc85d4a28 2022-03-29T17:49:42.299 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.metadata_list flags 1 0x7febc85e5d62 2022-03-29T17:49:42.299 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.metadata_set flags 3 0x7febc85ee5a7 2022-03-29T17:49:42.299 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.metadata_remove flags 3 0x7febc85cde87 2022-03-29T17:49:42.299 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.metadata_get flags 1 0x7febc85ce065 2022-03-29T17:49:42.299 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snapshot_get_limit flags 1 0x7febc85d0731 2022-03-29T17:49:42.300 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snapshot_set_limit flags 3 0x7febc85d3079 2022-03-29T17:49:42.300 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.child_attach flags 3 0x7febc85f473a 2022-03-29T17:49:42.300 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.child_detach flags 3 0x7febc85f60fc 2022-03-29T17:49:42.300 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.children_list flags 1 0x7febc85f0329 2022-03-29T17:49:42.300 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.migration_set flags 3 0x7febc85d5b72 2022-03-29T17:49:42.301 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.migration_set_state flags 3 0x7febc85d5dc9 2022-03-29T17:49:42.301 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.migration_get flags 1 0x7febc85d2075 2022-03-29T17:49:42.301 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.migration_remove flags 3 0x7febc85d4125 2022-03-29T17:49:42.301 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_modify_timestamp flags 3 0x7febc85d09aa 2022-03-29T17:49:42.301 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_access_timestamp flags 3 0x7febc85d08a5 2022-03-29T17:49:42.302 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.add_child flags 3 0x7febc85f8af5 2022-03-29T17:49:42.302 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.remove_child flags 3 0x7febc85f9052 2022-03-29T17:49:42.302 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_children flags 1 0x7febc85f9633 2022-03-29T17:49:42.303 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.get_id flags 1 0x7febc85c948d 2022-03-29T17:49:42.303 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.set_id flags 3 0x7febc85c96f0 2022-03-29T17:49:42.303 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.dir_get_id flags 1 0x7febc85ccc9d 2022-03-29T17:49:42.304 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.dir_get_name flags 1 0x7febc85cd59c 2022-03-29T17:49:42.304 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.dir_list flags 1 0x7febc85e35f8 2022-03-29T17:49:42.304 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.dir_add_image flags 3 0x7febc85e7a41 2022-03-29T17:49:42.305 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.dir_remove_image flags 3 0x7febc85cd41b 2022-03-29T17:49:42.305 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.dir_rename_image flags 3 0x7febc85e780d 2022-03-29T17:49:42.305 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.dir_state_assert flags 1 0x7febc85d033d 2022-03-29T17:49:42.305 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.dir_state_set flags 3 0x7febc85d2df0 2022-03-29T17:49:42.305 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.object_map_load flags 1 0x7febc85e7dee 2022-03-29T17:49:42.306 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.object_map_save flags 3 0x7febc85e82fc 2022-03-29T17:49:42.306 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.object_map_resize flags 3 0x7febc85e9a9b 2022-03-29T17:49:42.306 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.object_map_update flags 3 0x7febc85e8bce 2022-03-29T17:49:42.306 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.object_map_snap_add flags 3 0x7febc85e7ec6 2022-03-29T17:49:42.307 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.object_map_snap_remove flags 3 0x7febc85e84bf 2022-03-29T17:49:42.307 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snap_list flags 1 0x7febc85cff91 2022-03-29T17:49:42.307 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snap_add flags 3 0x7febc85d268e 2022-03-29T17:49:42.307 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snap_remove flags 3 0x7febc85c9b41 2022-03-29T17:49:42.307 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.snap_rename flags 3 0x7febc85ca2a2 2022-03-29T17:49:42.308 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_uuid_get flags 1 0x7febc85cfcef 2022-03-29T17:49:42.308 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_uuid_set flags 3 0x7febc85dc253 2022-03-29T17:49:42.308 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_mode_get flags 1 0x7febc85cac4b 2022-03-29T17:49:42.308 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_mode_set flags 3 0x7febc85dfba6 2022-03-29T17:49:42.309 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_peer_ping flags 3 0x7febc85ee3fb 2022-03-29T17:49:42.309 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_peer_list flags 1 0x7febc85e0035 2022-03-29T17:49:42.309 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_peer_add flags 3 0x7febc85e12de 2022-03-29T17:49:42.309 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_peer_remove flags 3 0x7febc85d813d 2022-03-29T17:49:42.309 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_client flags 3 0x7febc85d7b04 2022-03-29T17:49:42.310 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_cluster flags 3 0x7febc85e018f 2022-03-29T17:49:42.310 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_direction flags 3 0x7febc85d7ddb 2022-03-29T17:49:42.310 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_list flags 1 0x7febc85e2cae 2022-03-29T17:49:42.310 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_get_image_id flags 1 0x7febc85d8e8c 2022-03-29T17:49:42.311 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_get flags 1 0x7febc85d849a 2022-03-29T17:49:42.311 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_set flags 3 0x7febc85d8d0f 2022-03-29T17:49:42.311 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_remove flags 3 0x7febc85def25 2022-03-29T17:49:42.311 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_status_set flags 11 0x7febc85de7da 2022-03-29T17:49:42.311 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove flags 3 0x7febc85df02f 2022-03-29T17:49:42.312 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get flags 1 0x7febc85e0e5f 2022-03-29T17:49:42.312 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_status_list flags 1 0x7febc85ead7f 2022-03-29T17:49:42.312 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get_summary flags 1 0x7febc85eb570 2022-03-29T17:49:42.312 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove_down flags 3 0x7febc85ddf42 2022-03-29T17:49:42.313 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,229 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_get flags 1 0x7febc85de2cd 2022-03-29T17:49:42.313 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_list flags 1 0x7febc85ec04d 2022-03-29T17:49:42.313 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_instances_list flags 1 0x7febc85dc851 2022-03-29T17:49:42.314 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_instances_add flags 11 0x7febc85d94de 2022-03-29T17:49:42.314 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_instances_remove flags 3 0x7febc85d96ef 2022-03-29T17:49:42.315 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_image_map_list flags 1 0x7febc85dc05e 2022-03-29T17:49:42.315 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_image_map_update flags 2 0x7febc85d9841 2022-03-29T17:49:42.316 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_image_map_remove flags 2 0x7febc85d9bbd 2022-03-29T17:49:42.316 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_unlink_peer flags 3 0x7febc85fab3c 2022-03-29T17:49:42.316 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_set_copy_progress flags 3 0x7febc85f9e49 2022-03-29T17:49:42.316 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_dir_list flags 1 0x7febc85e2196 2022-03-29T17:49:42.317 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_dir_add flags 3 0x7febc85e6ca7 2022-03-29T17:49:42.317 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_dir_remove flags 3 0x7febc85cdcc2 2022-03-29T17:49:42.317 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_dir_rename flags 3 0x7febc85e6ec2 2022-03-29T17:49:42.317 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_image_remove flags 3 0x7febc85c732a 2022-03-29T17:49:42.317 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_image_list flags 1 0x7febc85dcc9a 2022-03-29T17:49:42.318 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_image_set flags 3 0x7febc85cacb3 2022-03-29T17:49:42.318 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.image_group_add flags 3 0x7febc85d4b03 2022-03-29T17:49:42.318 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.image_group_remove flags 3 0x7febc85d4f46 2022-03-29T17:49:42.318 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.image_group_get flags 1 0x7febc85d0c65 2022-03-29T17:49:42.319 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_snap_set flags 3 0x7febc85ecae6 2022-03-29T17:49:42.319 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_snap_remove flags 3 0x7febc85ceb76 2022-03-29T17:49:42.319 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_snap_get_by_id flags 1 0x7febc85ced09 2022-03-29T17:49:42.319 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.group_snap_list flags 1 0x7febc85e1c44 2022-03-29T17:49:42.319 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.trash_add flags 3 0x7febc85e579b 2022-03-29T17:49:42.320 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.trash_remove flags 3 0x7febc85d9de3 2022-03-29T17:49:42.324 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.trash_list flags 1 0x7febc85ecece 2022-03-29T17:49:42.324 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.trash_get flags 1 0x7febc85da0e8 2022-03-29T17:49:42.324 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.trash_state_set flags 3 0x7febc85da36e 2022-03-29T17:49:42.325 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.namespace_add flags 3 0x7febc85da7a6 2022-03-29T17:49:42.325 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.namespace_remove flags 3 0x7febc85da9d1 2022-03-29T17:49:42.326 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.namespace_list flags 1 0x7febc85dabba 2022-03-29T17:49:42.326 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.copyup flags 3 0x7febc85c716b 2022-03-29T17:49:42.326 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.sparse_copyup flags 3 0x7febc85f7e66 2022-03-29T17:49:42.327 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.assert_snapc_seq flags 3 0x7febc85c721b 2022-03-29T17:49:42.327 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method rbd.sparsify flags 3 0x7febc85cb074 2022-03-29T17:49:42.328 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _load_class rbd success 2022-03-29T17:49:42.328 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - open_all_classes found queue 2022-03-29T17:49:42.328 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _get_class adding new class name queue 0x612000026c00 2022-03-29T17:49:42.328 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _load_class queue from /usr/lib64/rados-classes/libcls_queue.so 2022-03-29T17:49:42.328 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,230 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/queue/cls_queue.cc:125: Loaded queue class! 2022-03-29T17:49:42.329 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_class queue status 3 2022-03-29T17:49:42.329 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method queue.queue_init flags 3 0x7febc838b256 2022-03-29T17:49:42.329 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method queue.queue_get_capacity flags 1 0x7febc838a9e9 2022-03-29T17:49:42.329 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method queue.queue_enqueue flags 3 0x7febc838b52e 2022-03-29T17:49:42.330 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method queue.queue_list_entries flags 1 0x7febc838adaf 2022-03-29T17:49:42.330 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method queue.queue_remove_entries flags 3 0x7febc838ab81 2022-03-29T17:49:42.330 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _load_class queue success 2022-03-29T17:49:42.330 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - open_all_classes found user 2022-03-29T17:49:42.331 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _get_class adding new class name user 0x612000026f00 2022-03-29T17:49:42.331 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _load_class user from /usr/lib64/rados-classes/libcls_user.so 2022-03-29T17:49:42.331 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,230 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/user/cls_user.cc:506: Loaded user class! 2022-03-29T17:49:42.331 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_class user status 3 2022-03-29T17:49:42.331 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method user.set_buckets_info flags 3 0x7febc8171c4b 2022-03-29T17:49:42.332 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method user.complete_stats_sync flags 3 0x7febc81719a3 2022-03-29T17:49:42.332 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method user.remove_bucket flags 3 0x7febc81711be 2022-03-29T17:49:42.332 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method user.list_buckets flags 1 0x7febc8172733 2022-03-29T17:49:42.332 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method user.get_header flags 1 0x7febc8170fc7 2022-03-29T17:49:42.333 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method user.reset_user_stats flags 3 0x7febc817054c 2022-03-29T17:49:42.333 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - register_cxx_method user.reset_user_stats2 flags 3 0x7febc817319b 2022-03-29T17:49:42.333 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _load_class user success 2022-03-29T17:49:42.333 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - open_all_classes found cas 2022-03-29T17:49:42.333 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _get_class adding new class name cas 0x612000027380 2022-03-29T17:49:42.334 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,230 [shard 0] osd - _load_class cas from /usr/lib64/rados-classes/libcls_cas.so 2022-03-29T17:49:42.334 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,231 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cas/cls_cas.cc:211: Loaded cas class! 2022-03-29T17:49:42.334 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_class cas status 3 2022-03-29T17:49:42.335 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method cas.chunk_create_or_get_ref flags 3 0x7febc7f5ab38 2022-03-29T17:49:42.335 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method cas.chunk_get_ref flags 3 0x7febc7f5a848 2022-03-29T17:49:42.335 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method cas.chunk_put_ref flags 3 0x7febc7f5a4c4 2022-03-29T17:49:42.335 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method cas.references_chunk flags 1 0x7febc7f5a1fc 2022-03-29T17:49:42.336 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _load_class cas success 2022-03-29T17:49:42.336 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - open_all_classes found fifo 2022-03-29T17:49:42.336 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _get_class adding new class name fifo 0x612000027680 2022-03-29T17:49:42.336 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _load_class fifo from /usr/lib64/rados-classes/libcls_fifo.so 2022-03-29T17:49:42.337 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_class fifo status 3 2022-03-29T17:49:42.337 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method fifo.create_meta flags 3 0x7febc7d1bdee 2022-03-29T17:49:42.337 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method fifo.get_meta flags 1 0x7febc7d1d0b2 2022-03-29T17:49:42.337 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method fifo.update_meta flags 3 0x7febc7d1d57b 2022-03-29T17:49:42.337 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method fifo.init_part flags 3 0x7febc7d1acf2 2022-03-29T17:49:42.338 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method fifo.push_part flags 3 0x7febc7d1b3e5 2022-03-29T17:49:42.338 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method fifo.trim_part flags 3 0x7febc7d1a782 2022-03-29T17:49:42.338 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method fifo.part_list flags 1 0x7febc7d19ed9 2022-03-29T17:49:42.338 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method fifo.get_part_info flags 1 0x7febc7d1977e 2022-03-29T17:49:42.339 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _load_class fifo success 2022-03-29T17:49:42.339 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - open_all_classes found cmpomap 2022-03-29T17:49:42.339 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _get_class adding new class name cmpomap 0x612000027980 2022-03-29T17:49:42.339 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _load_class cmpomap from /usr/lib64/rados-classes/libcls_cmpomap.so 2022-03-29T17:49:42.339 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,231 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cmpomap/server.cc:287: Loaded cmpomap class! 2022-03-29T17:49:42.340 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_class cmpomap status 3 2022-03-29T17:49:42.340 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method cmpomap.cmp_vals flags 1 0x7febc7afe2e7 2022-03-29T17:49:42.340 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method cmpomap.cmp_set_vals flags 3 0x7febc7afd70e 2022-03-29T17:49:42.340 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - register_cxx_method cmpomap.cmp_rm_keys flags 3 0x7febc7afd136 2022-03-29T17:49:42.340 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _load_class cmpomap success 2022-03-29T17:49:42.341 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - open_all_classes found hello 2022-03-29T17:49:42.341 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _get_class adding new class name hello 0x612000027e00 2022-03-29T17:49:42.341 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,231 [shard 0] osd - _load_class hello from /usr/lib64/rados-classes/libcls_hello.so 2022-03-29T17:49:42.341 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:42,232 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/hello/cls_hello.cc:316: loading cls_hello 2022-03-29T17:49:42.342 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_class hello status 3 2022-03-29T17:49:42.342 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.say_hello flags 1 0x7febc78edbf6 2022-03-29T17:49:42.342 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.record_hello flags 10 0x7febc78edea9 2022-03-29T17:49:42.342 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.write_return_data flags 2 0x7febc78edd7f 2022-03-29T17:49:42.342 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.writes_dont_return_data flags 2 0x7febc78edd7f 2022-03-29T17:49:42.343 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.write_too_much_return_data flags 2 0x7febc78edc6d 2022-03-29T17:49:42.343 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.replay flags 1 0x7febc78edbce 2022-03-29T17:49:42.343 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.turn_it_to_11 flags 11 0x7febc78ee808 2022-03-29T17:49:42.343 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.bad_reader flags 2 0x7febc78edbaf 2022-03-29T17:49:42.344 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method hello.bad_writer flags 1 0x7febc78edb9d 2022-03-29T17:49:42.344 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _load_class hello success 2022-03-29T17:49:42.344 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - open_all_classes found log 2022-03-29T17:49:42.344 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _get_class adding new class name log 0x612000028100 2022-03-29T17:49:42.345 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _load_class log from /usr/lib64/rados-classes/libcls_log.so 2022-03-29T17:49:42.345 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,232 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/log/cls_log.cc:305: Loaded log class! 2022-03-29T17:49:42.345 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_class log status 3 2022-03-29T17:49:42.345 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method log.add flags 3 0x7febc76dcf1f 2022-03-29T17:49:42.346 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method log.list flags 1 0x7febc76dda5d 2022-03-29T17:49:42.346 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method log.trim flags 3 0x7febc76dd2cf 2022-03-29T17:49:42.346 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method log.info flags 1 0x7febc76dcaf7 2022-03-29T17:49:42.346 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _load_class log success 2022-03-29T17:49:42.347 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - open_all_classes found otp 2022-03-29T17:49:42.347 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _get_class adding new class name otp 0x612000028580 2022-03-29T17:49:42.347 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _load_class otp from /usr/lib64/rados-classes/libcls_otp.so 2022-03-29T17:49:42.347 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_class otp status 3 2022-03-29T17:49:42.348 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method otp.otp_set flags 3 0x7febc74c937f 2022-03-29T17:49:42.348 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method otp.otp_get flags 1 0x7febc74c898f 2022-03-29T17:49:42.348 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method otp.otp_check flags 3 0x7febc74c7fc3 2022-03-29T17:49:42.348 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method otp.otp_get_result flags 1 0x7febc74c7902 2022-03-29T17:49:42.348 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method otp.otp_remove flags 3 0x7febc74c8700 2022-03-29T17:49:42.349 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method otp.get_current_time flags 1 0x7febc74c75f0 2022-03-29T17:49:42.349 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _load_class otp success 2022-03-29T17:49:42.349 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - open_all_classes found sdk 2022-03-29T17:49:42.349 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:42,232 [shard 0] osd - _get_class not permitted to load sdk 2022-03-29T17:49:42.349 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - open_all_classes found timeindex 2022-03-29T17:49:42.350 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _get_class adding new class name timeindex 0x612000028700 2022-03-29T17:49:42.350 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - _load_class timeindex from /usr/lib64/rados-classes/libcls_timeindex.so 2022-03-29T17:49:42.350 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,232 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/timeindex/cls_timeindex.cc:247: Loaded timeindex class! 2022-03-29T17:49:42.350 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_class timeindex status 3 2022-03-29T17:49:42.351 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method timeindex.add flags 3 0x7febc709a894 2022-03-29T17:49:42.351 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,232 [shard 0] osd - register_cxx_method timeindex.list flags 1 0x7febc709ac23 2022-03-29T17:49:42.351 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method timeindex.trim flags 3 0x7febc709a43d 2022-03-29T17:49:42.351 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _load_class timeindex success 2022-03-29T17:49:42.352 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - open_all_classes found lock 2022-03-29T17:49:42.352 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _get_class adding new class name lock 0x612000028b80 2022-03-29T17:49:42.352 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _load_class lock from /usr/lib64/rados-classes/libcls_lock.so 2022-03-29T17:49:42.352 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_class lock status 3 2022-03-29T17:49:42.352 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method lock.lock flags 11 0x7febc6e874af 2022-03-29T17:49:42.353 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method lock.unlock flags 11 0x7febc6e84e27 2022-03-29T17:49:42.353 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method lock.break_lock flags 3 0x7febc6e84ccd 2022-03-29T17:49:42.353 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method lock.get_info flags 1 0x7febc6e85a33 2022-03-29T17:49:42.353 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method lock.list_locks flags 1 0x7febc6e8326b 2022-03-29T17:49:42.354 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method lock.assert_locked flags 9 0x7febc6e843bd 2022-03-29T17:49:42.354 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method lock.set_cookie flags 11 0x7febc6e84ff4 2022-03-29T17:49:42.354 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _load_class lock success 2022-03-29T17:49:42.354 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - open_all_classes found 2pc_queue 2022-03-29T17:49:42.355 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _get_class adding new class name 2pc_queue 0x612000028d00 2022-03-29T17:49:42.355 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _load_class 2pc_queue from /usr/lib64/rados-classes/libcls_2pc_queue.so 2022-03-29T17:49:42.355 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,233 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/2pc_queue/cls_2pc_queue.cc:575: Loaded 2pc queue class! 2022-03-29T17:49:42.356 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_class 2pc_queue status 3 2022-03-29T17:49:42.356 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_init flags 3 0x7febc6c466a3 2022-03-29T17:49:42.357 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_get_capacity flags 1 0x7febc6c45c19 2022-03-29T17:49:42.357 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_reserve flags 3 0x7febc6c49504 2022-03-29T17:49:42.357 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_commit flags 3 0x7febc6c46b0d 2022-03-29T17:49:42.358 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_abort flags 3 0x7febc6c4828f 2022-03-29T17:49:42.358 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_reservations flags 1 0x7febc6c48c8c 2022-03-29T17:49:42.358 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_entries flags 1 0x7febc6c46076 2022-03-29T17:49:42.359 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_remove_entries flags 3 0x7febc6c45e39 2022-03-29T17:49:42.359 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_expire_reservations flags 3 0x7febc6c477d0 2022-03-29T17:49:42.359 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _load_class 2pc_queue success 2022-03-29T17:49:42.359 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - open_all_classes found refcount 2022-03-29T17:49:42.359 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _get_class adding new class name refcount 0x612000029000 2022-03-29T17:49:42.360 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,233 [shard 0] osd - _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so 2022-03-29T17:49:42.360 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,235 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/refcount/cls_refcount.cc:199: Loaded refcount class! 2022-03-29T17:49:42.360 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_class refcount status 3 2022-03-29T17:49:42.360 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method refcount.get flags 3 0x7febc697ec89 2022-03-29T17:49:42.361 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method refcount.put flags 3 0x7febc697e94c 2022-03-29T17:49:42.361 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method refcount.set flags 3 0x7febc697e0c1 2022-03-29T17:49:42.361 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method refcount.read flags 1 0x7febc697e602 2022-03-29T17:49:42.362 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - _load_class refcount success 2022-03-29T17:49:42.362 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - open_all_classes found version 2022-03-29T17:49:42.362 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - _get_class adding new class name version 0x612000029300 2022-03-29T17:49:42.362 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - _load_class version from /usr/lib64/rados-classes/libcls_version.so 2022-03-29T17:49:42.362 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,235 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/version/cls_version.cc:218: Loaded version class! 2022-03-29T17:49:42.363 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_class version status 3 2022-03-29T17:49:42.363 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method version.set flags 3 0x7febc66cd3d7 2022-03-29T17:49:42.363 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method version.inc flags 3 0x7febc66cd6e1 2022-03-29T17:49:42.364 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method version.inc_conds flags 3 0x7febc66cd6e1 2022-03-29T17:49:42.364 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method version.read flags 1 0x7febc66cd21c 2022-03-29T17:49:42.364 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method version.check_conds flags 1 0x7febc66cd4f9 2022-03-29T17:49:42.364 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - _load_class version success 2022-03-29T17:49:42.365 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - open_all_classes found rgw_gc 2022-03-29T17:49:42.365 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - _get_class adding new class name rgw_gc 0x612000029780 2022-03-29T17:49:42.365 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - _load_class rgw_gc from /usr/lib64/rados-classes/libcls_rgw_gc.so 2022-03-29T17:49:42.365 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,235 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw_gc/cls_rgw_gc.cc:539: Loaded rgw gc class! 2022-03-29T17:49:42.365 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_class rgw_gc status 3 2022-03-29T17:49:42.366 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_init flags 3 0x7febc6490a31 2022-03-29T17:49:42.366 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_enqueue flags 3 0x7febc6490130 2022-03-29T17:49:42.366 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_list_entries flags 1 0x7febc64922eb 2022-03-29T17:49:42.366 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_remove_entries flags 3 0x7febc6490d97 2022-03-29T17:49:42.367 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_update_entry flags 3 0x7febc6493e0e 2022-03-29T17:49:42.367 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:42,235 [shard 0] osd - _load_class rgw_gc success 2022-03-29T17:49:42.367 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:42,235 [shard 0] osd - OSD: nonce is 33163 2022-03-29T17:49:44.669 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:44,667 [shard 0] osd - _write_superblock writing superblock cluster_fsid 3be68405-f255-4011-9a3b-ba646329d493 osd_fsid 59cc9d2e-e40e-4048-b5cb-31db1b464c16 2022-03-29T17:49:44.670 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:44,668 [shard 0] osd - OSD::_write_superblock: do_transaction... 2022-03-29T17:49:44.675 INFO:teuthology.orchestra.run.smithi138.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 3be68405-f255-4011-9a3b-ba646329d493 2022-03-29T17:49:44.676 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:44,674 [shard 0] osd - stop 2022-03-29T17:49:44.676 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:44,674 [shard 0] osd - prepared to stop 2022-03-29T17:49:44.676 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:44,674 [shard 0] osd - stop 2022-03-29T17:49:44.677 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:44,674 [shard 0] alienstore - umount 2022-03-29T17:49:45.186 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:45,184 [shard 0] monc - stop 2022-03-29T17:49:45.186 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:45,184 [shard 0] mgrc - stop 2022-03-29T17:49:45.692 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-03-29T17:49:45.719 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-03-29T17:49:46.542 INFO:teuthology.orchestra.run.smithi138.stdout:warn: added seastar option --smp 1 2022-03-29T17:49:46.556 INFO:teuthology.orchestra.run.smithi138.stderr:==33978==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2022-03-29T17:49:46.561 INFO:teuthology.orchestra.run.smithi138.stderr:==33978==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc01358000; bottom 0x7f7fea9b0000; size: 0x007c169a8000 (532955168768) 2022-03-29T17:49:46.562 INFO:teuthology.orchestra.run.smithi138.stderr:False positive error reports may follow 2022-03-29T17:49:46.562 INFO:teuthology.orchestra.run.smithi138.stderr:For details see https://github.com/google/sanitizers/issues/189 2022-03-29T17:49:46.608 INFO:teuthology.orchestra.run.smithi138.stdout:WARNING: debug mode. Not for benchmarking or production 2022-03-29T17:49:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,606 [shard 0] osd - open_all_classes 2022-03-29T17:49:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,606 [shard 0] osd - open_all_classes found journal 2022-03-29T17:49:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,606 [shard 0] osd - _get_class adding new class name journal 0x612000026000 2022-03-29T17:49:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,606 [shard 0] osd - _load_class journal from /usr/lib64/rados-classes/libcls_journal.so 2022-03-29T17:49:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_class journal status 3 2022-03-29T17:49:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.create flags 3 0x7f7fe94df7de 2022-03-29T17:49:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.get_order flags 1 0x7f7fe94df1de 2022-03-29T17:49:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.get_splay_width flags 1 0x7f7fe94df246 2022-03-29T17:49:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.get_pool_id flags 1 0x7f7fe94dfc38 2022-03-29T17:49:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.get_minimum_set flags 1 0x7f7fe94dfe60 2022-03-29T17:49:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.set_minimum_set flags 3 0x7f7fe94dffb6 2022-03-29T17:49:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.get_active_set flags 1 0x7f7fe94dfed2 2022-03-29T17:49:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.set_active_set flags 3 0x7f7fe94e0155 2022-03-29T17:49:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.get_client flags 1 0x7f7fe94e14e4 2022-03-29T17:49:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.client_register flags 3 0x7f7fe94e2c5e 2022-03-29T17:49:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.client_update_data flags 3 0x7f7fe94e17bf 2022-03-29T17:49:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.client_update_state flags 3 0x7f7fe94e1c9f 2022-03-29T17:49:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.client_unregister flags 3 0x7f7fe94e4e8a 2022-03-29T17:49:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.client_commit flags 3 0x7f7fe94e5832 2022-03-29T17:49:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.client_list flags 1 0x7f7fe94e330f 2022-03-29T17:49:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.get_next_tag_tid flags 1 0x7f7fe94dff44 2022-03-29T17:49:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.get_tag flags 1 0x7f7fe94e0f5d 2022-03-29T17:49:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.tag_create flags 3 0x7f7fe94e5129 2022-03-29T17:49:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.tag_list flags 1 0x7f7fe94e3573 2022-03-29T17:49:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.guard_append flags 3 0x7f7fe94deb5c 2022-03-29T17:49:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - register_cxx_method journal.append flags 3 0x7f7fe94e02f4 2022-03-29T17:49:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - _load_class journal success 2022-03-29T17:49:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - open_all_classes found rgw 2022-03-29T17:49:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - _get_class adding new class name rgw 0x612000026180 2022-03-29T17:49:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,607 [shard 0] osd - _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so 2022-03-29T17:49:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,608 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw/cls_rgw.cc:4353: Loaded rgw class! 2022-03-29T17:49:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,608 [shard 0] osd - register_class rgw status 3 2022-03-29T17:49:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,608 [shard 0] osd - register_cxx_method rgw.bucket_init_index flags 3 0x7f7fe9194175 2022-03-29T17:49:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,608 [shard 0] osd - register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f7fe91a3ca1 2022-03-29T17:49:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,608 [shard 0] osd - register_cxx_method rgw.bucket_list flags 1 0x7f7fe91b0331 2022-03-29T17:49:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,608 [shard 0] osd - register_cxx_method rgw.bucket_check_index flags 1 0x7f7fe91aeea4 2022-03-29T17:49:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f7fe91af70f 2022-03-29T17:49:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_update_stats flags 3 0x7f7fe91a55bd 2022-03-29T17:49:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_prepare_op flags 3 0x7f7fe919f3dd 2022-03-29T17:49:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_complete_op flags 3 0x7f7fe91a725b 2022-03-29T17:49:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_link_olh flags 3 0x7f7fe91b249b 2022-03-29T17:49:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f7fe91a9434 2022-03-29T17:49:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f7fe91aad50 2022-03-29T17:49:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f7fe91a8dcb 2022-03-29T17:49:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bucket_clear_olh flags 3 0x7f7fe91a861e 2022-03-29T17:49:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.obj_remove flags 3 0x7f7fe91ac48a 2022-03-29T17:49:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.obj_store_pg_ver flags 2 0x7f7fe9198019 2022-03-29T17:49:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f7fe9197cda 2022-03-29T17:49:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.obj_check_mtime flags 1 0x7f7fe9197a18 2022-03-29T17:49:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bi_get flags 1 0x7f7fe91974fe 2022-03-29T17:49:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bi_put flags 3 0x7f7fe9197304 2022-03-29T17:49:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bi_list flags 1 0x7f7fe91adb1e 2022-03-29T17:49:46.666 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bi_log_list flags 1 0x7f7fe919ef1a 2022-03-29T17:49:46.666 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bi_log_trim flags 3 0x7f7fe9196d56 2022-03-29T17:49:46.666 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.dir_suggest_changes flags 3 0x7f7fe91a593a 2022-03-29T17:49:46.666 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bi_log_resync flags 3 0x7f7fe91a4fe3 2022-03-29T17:49:46.666 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.bi_log_stop flags 3 0x7f7fe91a4a09 2022-03-29T17:49:46.667 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.user_usage_log_add flags 3 0x7f7fe91a0e48 2022-03-29T17:49:46.667 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.user_usage_log_read flags 1 0x7f7fe91a06db 2022-03-29T17:49:46.667 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.user_usage_log_trim flags 3 0x7f7fe91a0bf9 2022-03-29T17:49:46.667 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.usage_log_clear flags 2 0x7f7fe9191dda 2022-03-29T17:49:46.668 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.gc_set_entry flags 3 0x7f7fe919a38e 2022-03-29T17:49:46.668 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.gc_defer_entry flags 3 0x7f7fe919a244 2022-03-29T17:49:46.668 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.gc_list flags 1 0x7f7fe91989c9 2022-03-29T17:49:46.668 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.gc_remove flags 3 0x7f7fe91afeb9 2022-03-29T17:49:46.669 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.lc_get_entry flags 1 0x7f7fe9199423 2022-03-29T17:49:46.669 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.lc_set_entry flags 3 0x7f7fe919b271 2022-03-29T17:49:46.669 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.lc_rm_entry flags 3 0x7f7fe919b122 2022-03-29T17:49:46.669 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.lc_get_next_entry flags 1 0x7f7fe9196850 2022-03-29T17:49:46.670 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.lc_put_head flags 3 0x7f7fe91965fb 2022-03-29T17:49:46.670 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.lc_get_head flags 1 0x7f7fe91962c1 2022-03-29T17:49:46.670 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.lc_list_entries flags 1 0x7f7fe919d3ee 2022-03-29T17:49:46.670 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.reshard_add flags 3 0x7f7fe9195cbc 2022-03-29T17:49:46.670 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.reshard_list flags 1 0x7f7fe919a941 2022-03-29T17:49:46.671 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.reshard_get flags 1 0x7f7fe91955af 2022-03-29T17:49:46.671 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.reshard_remove flags 3 0x7f7fe9195188 2022-03-29T17:49:46.671 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.set_bucket_resharding flags 3 0x7f7fe91a474a 2022-03-29T17:49:46.671 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.clear_bucket_resharding flags 3 0x7f7fe91a44ee 2022-03-29T17:49:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.guard_bucket_resharding flags 1 0x7f7fe91a4291 2022-03-29T17:49:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method rgw.get_bucket_resharding flags 1 0x7f7fe91a3eea 2022-03-29T17:49:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _load_class rgw success 2022-03-29T17:49:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - open_all_classes found cephfs 2022-03-29T17:49:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _get_class adding new class name cephfs 0x612000026480 2022-03-29T17:49:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _load_class cephfs from /usr/lib64/rados-classes/libcls_cephfs.so 2022-03-29T17:49:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:46,609 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cephfs/cls_cephfs.cc:201: loading cephfs 2022-03-29T17:49:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_class cephfs status 3 2022-03-29T17:49:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method cephfs.accumulate_inode_metadata flags 3 0x7f7fe8edf2f9 2022-03-29T17:49:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _load_class cephfs success 2022-03-29T17:49:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - open_all_classes found numops 2022-03-29T17:49:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _get_class adding new class name numops 0x612000026780 2022-03-29T17:49:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _load_class numops from /usr/lib64/rados-classes/libcls_numops.so 2022-03-29T17:49:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_class numops status 3 2022-03-29T17:49:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method numops.add flags 3 0x7f7fe8cd165f 2022-03-29T17:49:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - register_cxx_method numops.mul flags 3 0x7f7fe8cd0be6 2022-03-29T17:49:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _load_class numops success 2022-03-29T17:49:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - open_all_classes found lua 2022-03-29T17:49:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:46,609 [shard 0] osd - _get_class not permitted to load lua 2022-03-29T17:49:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - open_all_classes found test_remote_reads 2022-03-29T17:49:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:46,609 [shard 0] osd - _get_class not permitted to load test_remote_reads 2022-03-29T17:49:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - open_all_classes found rbd 2022-03-29T17:49:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _get_class adding new class name rbd 0x612000026a80 2022-03-29T17:49:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,609 [shard 0] osd - _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so 2022-03-29T17:49:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_class rbd status 3 2022-03-29T17:49:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.create flags 3 0x7f7fe8a71ec0 2022-03-29T17:49:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_features flags 1 0x7f7fe8a5f5af 2022-03-29T17:49:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_features flags 3 0x7f7fe8a5f7d2 2022-03-29T17:49:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_size flags 1 0x7f7fe8a84cb9 2022-03-29T17:49:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_size flags 3 0x7f7fe8a6475d 2022-03-29T17:49:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_snapcontext flags 1 0x7f7fe8a6027a 2022-03-29T17:49:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_object_prefix flags 1 0x7f7fe8a5eaaf 2022-03-29T17:49:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_data_pool flags 1 0x7f7fe8a5e5be 2022-03-29T17:49:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_snapshot_name flags 1 0x7f7fe8a7df69 2022-03-29T17:49:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_snapshot_timestamp flags 1 0x7f7fe8a7dbc7 2022-03-29T17:49:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snapshot_get flags 1 0x7f7fe8a82dcd 2022-03-29T17:49:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snapshot_add flags 3 0x7f7fe8a899a5 2022-03-29T17:49:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snapshot_remove flags 3 0x7f7fe8a820e4 2022-03-29T17:49:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snapshot_rename flags 3 0x7f7fe8a7f938 2022-03-29T17:49:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snapshot_trash_add flags 3 0x7f7fe8a7e82f 2022-03-29T17:49:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_all_features flags 1 0x7f7fe8a550bc 2022-03-29T17:49:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_parent flags 1 0x7f7fe8a7cd9f 2022-03-29T17:49:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_parent flags 3 0x7f7fe8a6f007 2022-03-29T17:49:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.remove_parent flags 3 0x7f7fe8a81a52 2022-03-29T17:49:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.parent_get flags 1 0x7f7fe8a7d91f 2022-03-29T17:49:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.parent_overlap_get flags 1 0x7f7fe8a85270 2022-03-29T17:49:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.parent_attach flags 3 0x7f7fe8a65489 2022-03-29T17:49:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.parent_detach flags 3 0x7f7fe8a81a72 2022-03-29T17:49:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_protection_status flags 3 0x7f7fe8a847f9 2022-03-29T17:49:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_protection_status flags 1 0x7f7fe8a83655 2022-03-29T17:49:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f7fe8a62140 2022-03-29T17:49:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f7fe8a615c9 2022-03-29T17:49:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_create_timestamp flags 1 0x7f7fe8a5ee65 2022-03-29T17:49:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_access_timestamp flags 1 0x7f7fe8a5f0d3 2022-03-29T17:49:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_modify_timestamp flags 1 0x7f7fe8a5f341 2022-03-29T17:49:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_flags flags 1 0x7f7fe8a85842 2022-03-29T17:49:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_flags flags 3 0x7f7fe8a81a92 2022-03-29T17:49:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.op_features_get flags 1 0x7f7fe8a5e44f 2022-03-29T17:49:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.op_features_set flags 3 0x7f7fe8a62a28 2022-03-29T17:49:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.metadata_list flags 1 0x7f7fe8a73d62 2022-03-29T17:49:46.685 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.metadata_set flags 3 0x7f7fe8a7c5a7 2022-03-29T17:49:46.685 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.metadata_remove flags 3 0x7f7fe8a5be87 2022-03-29T17:49:46.685 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.metadata_get flags 1 0x7f7fe8a5c065 2022-03-29T17:49:46.685 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snapshot_get_limit flags 1 0x7f7fe8a5e731 2022-03-29T17:49:46.685 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snapshot_set_limit flags 3 0x7f7fe8a61079 2022-03-29T17:49:46.686 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.child_attach flags 3 0x7f7fe8a8273a 2022-03-29T17:49:46.686 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.child_detach flags 3 0x7f7fe8a840fc 2022-03-29T17:49:46.686 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.children_list flags 1 0x7f7fe8a7e329 2022-03-29T17:49:46.686 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.migration_set flags 3 0x7f7fe8a63b72 2022-03-29T17:49:46.686 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.migration_set_state flags 3 0x7f7fe8a63dc9 2022-03-29T17:49:46.687 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.migration_get flags 1 0x7f7fe8a60075 2022-03-29T17:49:46.687 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.migration_remove flags 3 0x7f7fe8a62125 2022-03-29T17:49:46.687 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_modify_timestamp flags 3 0x7f7fe8a5e9aa 2022-03-29T17:49:46.687 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_access_timestamp flags 3 0x7f7fe8a5e8a5 2022-03-29T17:49:46.688 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.add_child flags 3 0x7f7fe8a86af5 2022-03-29T17:49:46.688 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.remove_child flags 3 0x7f7fe8a87052 2022-03-29T17:49:46.688 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_children flags 1 0x7f7fe8a87633 2022-03-29T17:49:46.688 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.get_id flags 1 0x7f7fe8a5748d 2022-03-29T17:49:46.688 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.set_id flags 3 0x7f7fe8a576f0 2022-03-29T17:49:46.689 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.dir_get_id flags 1 0x7f7fe8a5ac9d 2022-03-29T17:49:46.689 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.dir_get_name flags 1 0x7f7fe8a5b59c 2022-03-29T17:49:46.689 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.dir_list flags 1 0x7f7fe8a715f8 2022-03-29T17:49:46.689 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.dir_add_image flags 3 0x7f7fe8a75a41 2022-03-29T17:49:46.689 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.dir_remove_image flags 3 0x7f7fe8a5b41b 2022-03-29T17:49:46.690 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.dir_rename_image flags 3 0x7f7fe8a7580d 2022-03-29T17:49:46.690 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.dir_state_assert flags 1 0x7f7fe8a5e33d 2022-03-29T17:49:46.690 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.dir_state_set flags 3 0x7f7fe8a60df0 2022-03-29T17:49:46.690 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.object_map_load flags 1 0x7f7fe8a75dee 2022-03-29T17:49:46.691 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.object_map_save flags 3 0x7f7fe8a762fc 2022-03-29T17:49:46.691 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.object_map_resize flags 3 0x7f7fe8a77a9b 2022-03-29T17:49:46.691 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.object_map_update flags 3 0x7f7fe8a76bce 2022-03-29T17:49:46.691 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.object_map_snap_add flags 3 0x7f7fe8a75ec6 2022-03-29T17:49:46.691 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.object_map_snap_remove flags 3 0x7f7fe8a764bf 2022-03-29T17:49:46.692 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snap_list flags 1 0x7f7fe8a5df91 2022-03-29T17:49:46.692 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snap_add flags 3 0x7f7fe8a6068e 2022-03-29T17:49:46.692 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snap_remove flags 3 0x7f7fe8a57b41 2022-03-29T17:49:46.692 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.snap_rename flags 3 0x7f7fe8a582a2 2022-03-29T17:49:46.692 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_uuid_get flags 1 0x7f7fe8a5dcef 2022-03-29T17:49:46.693 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_uuid_set flags 3 0x7f7fe8a6a253 2022-03-29T17:49:46.693 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_mode_get flags 1 0x7f7fe8a58c4b 2022-03-29T17:49:46.693 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_mode_set flags 3 0x7f7fe8a6dba6 2022-03-29T17:49:46.693 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_peer_ping flags 3 0x7f7fe8a7c3fb 2022-03-29T17:49:46.693 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_peer_list flags 1 0x7f7fe8a6e035 2022-03-29T17:49:46.694 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_peer_add flags 3 0x7f7fe8a6f2de 2022-03-29T17:49:46.694 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_peer_remove flags 3 0x7f7fe8a6613d 2022-03-29T17:49:46.694 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_client flags 3 0x7f7fe8a65b04 2022-03-29T17:49:46.694 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_cluster flags 3 0x7f7fe8a6e18f 2022-03-29T17:49:46.695 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_direction flags 3 0x7f7fe8a65ddb 2022-03-29T17:49:46.695 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_image_list flags 1 0x7f7fe8a70cae 2022-03-29T17:49:46.695 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_image_get_image_id flags 1 0x7f7fe8a66e8c 2022-03-29T17:49:46.695 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_image_get flags 1 0x7f7fe8a6649a 2022-03-29T17:49:46.695 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,610 [shard 0] osd - register_cxx_method rbd.mirror_image_set flags 3 0x7f7fe8a66d0f 2022-03-29T17:49:46.696 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_remove flags 3 0x7f7fe8a6cf25 2022-03-29T17:49:46.696 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_status_set flags 11 0x7f7fe8a6c7da 2022-03-29T17:49:46.696 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove flags 3 0x7f7fe8a6d02f 2022-03-29T17:49:46.696 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get flags 1 0x7f7fe8a6ee5f 2022-03-29T17:49:46.696 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_status_list flags 1 0x7f7fe8a78d7f 2022-03-29T17:49:46.697 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get_summary flags 1 0x7f7fe8a79570 2022-03-29T17:49:46.697 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove_down flags 3 0x7f7fe8a6bf42 2022-03-29T17:49:46.697 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_get flags 1 0x7f7fe8a6c2cd 2022-03-29T17:49:46.697 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_list flags 1 0x7f7fe8a7a04d 2022-03-29T17:49:46.698 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_instances_list flags 1 0x7f7fe8a6a851 2022-03-29T17:49:46.698 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_instances_add flags 11 0x7f7fe8a674de 2022-03-29T17:49:46.698 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_instances_remove flags 3 0x7f7fe8a676ef 2022-03-29T17:49:46.698 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_map_list flags 1 0x7f7fe8a6a05e 2022-03-29T17:49:46.698 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_map_update flags 2 0x7f7fe8a67841 2022-03-29T17:49:46.699 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_map_remove flags 2 0x7f7fe8a67bbd 2022-03-29T17:49:46.699 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_unlink_peer flags 3 0x7f7fe8a88b3c 2022-03-29T17:49:46.699 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_set_copy_progress flags 3 0x7f7fe8a87e49 2022-03-29T17:49:46.699 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_dir_list flags 1 0x7f7fe8a70196 2022-03-29T17:49:46.699 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_dir_add flags 3 0x7f7fe8a74ca7 2022-03-29T17:49:46.700 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_dir_remove flags 3 0x7f7fe8a5bcc2 2022-03-29T17:49:46.700 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_dir_rename flags 3 0x7f7fe8a74ec2 2022-03-29T17:49:46.700 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_image_remove flags 3 0x7f7fe8a5532a 2022-03-29T17:49:46.700 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_image_list flags 1 0x7f7fe8a6ac9a 2022-03-29T17:49:46.700 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_image_set flags 3 0x7f7fe8a58cb3 2022-03-29T17:49:46.701 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.image_group_add flags 3 0x7f7fe8a62b03 2022-03-29T17:49:46.701 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.image_group_remove flags 3 0x7f7fe8a62f46 2022-03-29T17:49:46.701 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.image_group_get flags 1 0x7f7fe8a5ec65 2022-03-29T17:49:46.701 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_snap_set flags 3 0x7f7fe8a7aae6 2022-03-29T17:49:46.702 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_snap_remove flags 3 0x7f7fe8a5cb76 2022-03-29T17:49:46.702 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_snap_get_by_id flags 1 0x7f7fe8a5cd09 2022-03-29T17:49:46.702 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.group_snap_list flags 1 0x7f7fe8a6fc44 2022-03-29T17:49:46.702 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.trash_add flags 3 0x7f7fe8a7379b 2022-03-29T17:49:46.702 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.trash_remove flags 3 0x7f7fe8a67de3 2022-03-29T17:49:46.703 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.trash_list flags 1 0x7f7fe8a7aece 2022-03-29T17:49:46.703 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.trash_get flags 1 0x7f7fe8a680e8 2022-03-29T17:49:46.703 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.trash_state_set flags 3 0x7f7fe8a6836e 2022-03-29T17:49:46.703 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.namespace_add flags 3 0x7f7fe8a687a6 2022-03-29T17:49:46.704 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.namespace_remove flags 3 0x7f7fe8a689d1 2022-03-29T17:49:46.704 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.namespace_list flags 1 0x7f7fe8a68bba 2022-03-29T17:49:46.704 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.copyup flags 3 0x7f7fe8a5516b 2022-03-29T17:49:46.704 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.sparse_copyup flags 3 0x7f7fe8a85e66 2022-03-29T17:49:46.704 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.assert_snapc_seq flags 3 0x7f7fe8a5521b 2022-03-29T17:49:46.705 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method rbd.sparsify flags 3 0x7f7fe8a59074 2022-03-29T17:49:46.705 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - _load_class rbd success 2022-03-29T17:49:46.705 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - open_all_classes found queue 2022-03-29T17:49:46.705 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - _get_class adding new class name queue 0x612000026c00 2022-03-29T17:49:46.705 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - _load_class queue from /usr/lib64/rados-classes/libcls_queue.so 2022-03-29T17:49:46.706 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,611 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/queue/cls_queue.cc:125: Loaded queue class! 2022-03-29T17:49:46.706 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_class queue status 3 2022-03-29T17:49:46.706 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method queue.queue_init flags 3 0x7f7fe8819256 2022-03-29T17:49:46.706 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method queue.queue_get_capacity flags 1 0x7f7fe88189e9 2022-03-29T17:49:46.707 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method queue.queue_enqueue flags 3 0x7f7fe881952e 2022-03-29T17:49:46.707 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method queue.queue_list_entries flags 1 0x7f7fe8818daf 2022-03-29T17:49:46.707 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method queue.queue_remove_entries flags 3 0x7f7fe8818b81 2022-03-29T17:49:46.707 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - _load_class queue success 2022-03-29T17:49:46.707 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - open_all_classes found user 2022-03-29T17:49:46.708 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - _get_class adding new class name user 0x612000026f00 2022-03-29T17:49:46.708 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - _load_class user from /usr/lib64/rados-classes/libcls_user.so 2022-03-29T17:49:46.708 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,611 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/user/cls_user.cc:506: Loaded user class! 2022-03-29T17:49:46.708 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_class user status 3 2022-03-29T17:49:46.708 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method user.set_buckets_info flags 3 0x7f7fe85ffc4b 2022-03-29T17:49:46.709 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method user.complete_stats_sync flags 3 0x7f7fe85ff9a3 2022-03-29T17:49:46.709 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method user.remove_bucket flags 3 0x7f7fe85ff1be 2022-03-29T17:49:46.709 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,611 [shard 0] osd - register_cxx_method user.list_buckets flags 1 0x7f7fe8600733 2022-03-29T17:49:46.709 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method user.get_header flags 1 0x7f7fe85fefc7 2022-03-29T17:49:46.710 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method user.reset_user_stats flags 3 0x7f7fe85fe54c 2022-03-29T17:49:46.710 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method user.reset_user_stats2 flags 3 0x7f7fe860119b 2022-03-29T17:49:46.710 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _load_class user success 2022-03-29T17:49:46.710 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - open_all_classes found cas 2022-03-29T17:49:46.710 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _get_class adding new class name cas 0x612000027380 2022-03-29T17:49:46.711 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _load_class cas from /usr/lib64/rados-classes/libcls_cas.so 2022-03-29T17:49:46.711 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,612 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cas/cls_cas.cc:211: Loaded cas class! 2022-03-29T17:49:46.711 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_class cas status 3 2022-03-29T17:49:46.711 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method cas.chunk_create_or_get_ref flags 3 0x7f7fe83e8b38 2022-03-29T17:49:46.711 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method cas.chunk_get_ref flags 3 0x7f7fe83e8848 2022-03-29T17:49:46.712 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method cas.chunk_put_ref flags 3 0x7f7fe83e84c4 2022-03-29T17:49:46.712 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method cas.references_chunk flags 1 0x7f7fe83e81fc 2022-03-29T17:49:46.712 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _load_class cas success 2022-03-29T17:49:46.712 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - open_all_classes found fifo 2022-03-29T17:49:46.713 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _get_class adding new class name fifo 0x612000027680 2022-03-29T17:49:46.713 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _load_class fifo from /usr/lib64/rados-classes/libcls_fifo.so 2022-03-29T17:49:46.713 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_class fifo status 3 2022-03-29T17:49:46.713 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method fifo.create_meta flags 3 0x7f7fe81a9dee 2022-03-29T17:49:46.713 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method fifo.get_meta flags 1 0x7f7fe81ab0b2 2022-03-29T17:49:46.714 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method fifo.update_meta flags 3 0x7f7fe81ab57b 2022-03-29T17:49:46.714 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method fifo.init_part flags 3 0x7f7fe81a8cf2 2022-03-29T17:49:46.714 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method fifo.push_part flags 3 0x7f7fe81a93e5 2022-03-29T17:49:46.714 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method fifo.trim_part flags 3 0x7f7fe81a8782 2022-03-29T17:49:46.714 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method fifo.part_list flags 1 0x7f7fe81a7ed9 2022-03-29T17:49:46.715 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method fifo.get_part_info flags 1 0x7f7fe81a777e 2022-03-29T17:49:46.715 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _load_class fifo success 2022-03-29T17:49:46.715 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - open_all_classes found cmpomap 2022-03-29T17:49:46.715 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _get_class adding new class name cmpomap 0x612000027980 2022-03-29T17:49:46.715 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _load_class cmpomap from /usr/lib64/rados-classes/libcls_cmpomap.so 2022-03-29T17:49:46.716 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,612 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cmpomap/server.cc:287: Loaded cmpomap class! 2022-03-29T17:49:46.716 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_class cmpomap status 3 2022-03-29T17:49:46.716 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method cmpomap.cmp_vals flags 1 0x7f7fe7f8c2e7 2022-03-29T17:49:46.716 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method cmpomap.cmp_set_vals flags 3 0x7f7fe7f8b70e 2022-03-29T17:49:46.716 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - register_cxx_method cmpomap.cmp_rm_keys flags 3 0x7f7fe7f8b136 2022-03-29T17:49:46.717 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _load_class cmpomap success 2022-03-29T17:49:46.717 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - open_all_classes found hello 2022-03-29T17:49:46.717 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _get_class adding new class name hello 0x612000027e00 2022-03-29T17:49:46.717 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,612 [shard 0] osd - _load_class hello from /usr/lib64/rados-classes/libcls_hello.so 2022-03-29T17:49:46.718 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:46,613 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/hello/cls_hello.cc:316: loading cls_hello 2022-03-29T17:49:46.718 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_class hello status 3 2022-03-29T17:49:46.718 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.say_hello flags 1 0x7f7fe7d7bbf6 2022-03-29T17:49:46.718 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.record_hello flags 10 0x7f7fe7d7bea9 2022-03-29T17:49:46.718 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.write_return_data flags 2 0x7f7fe7d7bd7f 2022-03-29T17:49:46.719 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.writes_dont_return_data flags 2 0x7f7fe7d7bd7f 2022-03-29T17:49:46.719 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.write_too_much_return_data flags 2 0x7f7fe7d7bc6d 2022-03-29T17:49:46.719 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.replay flags 1 0x7f7fe7d7bbce 2022-03-29T17:49:46.719 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.turn_it_to_11 flags 11 0x7f7fe7d7c808 2022-03-29T17:49:46.720 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.bad_reader flags 2 0x7f7fe7d7bbaf 2022-03-29T17:49:46.720 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method hello.bad_writer flags 1 0x7f7fe7d7bb9d 2022-03-29T17:49:46.720 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _load_class hello success 2022-03-29T17:49:46.720 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - open_all_classes found log 2022-03-29T17:49:46.720 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _get_class adding new class name log 0x612000028100 2022-03-29T17:49:46.721 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _load_class log from /usr/lib64/rados-classes/libcls_log.so 2022-03-29T17:49:46.721 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,613 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/log/cls_log.cc:305: Loaded log class! 2022-03-29T17:49:46.721 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_class log status 3 2022-03-29T17:49:46.721 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method log.add flags 3 0x7f7fe7b6af1f 2022-03-29T17:49:46.721 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method log.list flags 1 0x7f7fe7b6ba5d 2022-03-29T17:49:46.722 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method log.trim flags 3 0x7f7fe7b6b2cf 2022-03-29T17:49:46.722 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method log.info flags 1 0x7f7fe7b6aaf7 2022-03-29T17:49:46.722 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _load_class log success 2022-03-29T17:49:46.722 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - open_all_classes found otp 2022-03-29T17:49:46.723 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _get_class adding new class name otp 0x612000028580 2022-03-29T17:49:46.723 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _load_class otp from /usr/lib64/rados-classes/libcls_otp.so 2022-03-29T17:49:46.723 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_class otp status 3 2022-03-29T17:49:46.723 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method otp.otp_set flags 3 0x7f7fe795737f 2022-03-29T17:49:46.723 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method otp.otp_get flags 1 0x7f7fe795698f 2022-03-29T17:49:46.724 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method otp.otp_check flags 3 0x7f7fe7955fc3 2022-03-29T17:49:46.724 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method otp.otp_get_result flags 1 0x7f7fe7955902 2022-03-29T17:49:46.724 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method otp.otp_remove flags 3 0x7f7fe7956700 2022-03-29T17:49:46.724 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method otp.get_current_time flags 1 0x7f7fe79555f0 2022-03-29T17:49:46.724 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _load_class otp success 2022-03-29T17:49:46.725 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - open_all_classes found sdk 2022-03-29T17:49:46.725 INFO:teuthology.orchestra.run.smithi138.stderr:WARN 2022-03-29 17:49:46,613 [shard 0] osd - _get_class not permitted to load sdk 2022-03-29T17:49:46.725 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - open_all_classes found timeindex 2022-03-29T17:49:46.725 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _get_class adding new class name timeindex 0x612000028700 2022-03-29T17:49:46.725 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - _load_class timeindex from /usr/lib64/rados-classes/libcls_timeindex.so 2022-03-29T17:49:46.726 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,613 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/timeindex/cls_timeindex.cc:247: Loaded timeindex class! 2022-03-29T17:49:46.726 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_class timeindex status 3 2022-03-29T17:49:46.726 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,613 [shard 0] osd - register_cxx_method timeindex.add flags 3 0x7f7fe7528894 2022-03-29T17:49:46.726 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method timeindex.list flags 1 0x7f7fe7528c23 2022-03-29T17:49:46.727 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method timeindex.trim flags 3 0x7f7fe752843d 2022-03-29T17:49:46.727 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _load_class timeindex success 2022-03-29T17:49:46.727 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - open_all_classes found lock 2022-03-29T17:49:46.727 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _get_class adding new class name lock 0x612000028b80 2022-03-29T17:49:46.727 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _load_class lock from /usr/lib64/rados-classes/libcls_lock.so 2022-03-29T17:49:46.728 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_class lock status 3 2022-03-29T17:49:46.728 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method lock.lock flags 11 0x7f7fe73154af 2022-03-29T17:49:46.728 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method lock.unlock flags 11 0x7f7fe7312e27 2022-03-29T17:49:46.728 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method lock.break_lock flags 3 0x7f7fe7312ccd 2022-03-29T17:49:46.728 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method lock.get_info flags 1 0x7f7fe7313a33 2022-03-29T17:49:46.729 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method lock.list_locks flags 1 0x7f7fe731126b 2022-03-29T17:49:46.729 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method lock.assert_locked flags 9 0x7f7fe73123bd 2022-03-29T17:49:46.729 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method lock.set_cookie flags 11 0x7f7fe7312ff4 2022-03-29T17:49:46.729 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _load_class lock success 2022-03-29T17:49:46.730 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - open_all_classes found 2pc_queue 2022-03-29T17:49:46.730 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _get_class adding new class name 2pc_queue 0x612000028d00 2022-03-29T17:49:46.730 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _load_class 2pc_queue from /usr/lib64/rados-classes/libcls_2pc_queue.so 2022-03-29T17:49:46.730 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,614 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/2pc_queue/cls_2pc_queue.cc:575: Loaded 2pc queue class! 2022-03-29T17:49:46.730 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_class 2pc_queue status 3 2022-03-29T17:49:46.731 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_init flags 3 0x7f7fe70d46a3 2022-03-29T17:49:46.731 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_get_capacity flags 1 0x7f7fe70d3c19 2022-03-29T17:49:46.731 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_reserve flags 3 0x7f7fe70d7504 2022-03-29T17:49:46.731 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_commit flags 3 0x7f7fe70d4b0d 2022-03-29T17:49:46.731 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_abort flags 3 0x7f7fe70d628f 2022-03-29T17:49:46.732 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_reservations flags 1 0x7f7fe70d6c8c 2022-03-29T17:49:46.732 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_entries flags 1 0x7f7fe70d4076 2022-03-29T17:49:46.732 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_remove_entries flags 3 0x7f7fe70d3e39 2022-03-29T17:49:46.732 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_expire_reservations flags 3 0x7f7fe70d57d0 2022-03-29T17:49:46.732 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _load_class 2pc_queue success 2022-03-29T17:49:46.733 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - open_all_classes found refcount 2022-03-29T17:49:46.733 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _get_class adding new class name refcount 0x612000029000 2022-03-29T17:49:46.733 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,614 [shard 0] osd - _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so 2022-03-29T17:49:46.733 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,615 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/refcount/cls_refcount.cc:199: Loaded refcount class! 2022-03-29T17:49:46.734 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - register_class refcount status 3 2022-03-29T17:49:46.734 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - register_cxx_method refcount.get flags 3 0x7f7fe6e0cc89 2022-03-29T17:49:46.734 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - register_cxx_method refcount.put flags 3 0x7f7fe6e0c94c 2022-03-29T17:49:46.734 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - register_cxx_method refcount.set flags 3 0x7f7fe6e0c0c1 2022-03-29T17:49:46.734 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - register_cxx_method refcount.read flags 1 0x7f7fe6e0c602 2022-03-29T17:49:46.735 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - _load_class refcount success 2022-03-29T17:49:46.735 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - open_all_classes found version 2022-03-29T17:49:46.735 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - _get_class adding new class name version 0x612000029300 2022-03-29T17:49:46.735 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,615 [shard 0] osd - _load_class version from /usr/lib64/rados-classes/libcls_version.so 2022-03-29T17:49:46.735 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,616 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/version/cls_version.cc:218: Loaded version class! 2022-03-29T17:49:46.736 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_class version status 3 2022-03-29T17:49:46.736 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method version.set flags 3 0x7f7fe6b5b3d7 2022-03-29T17:49:46.736 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method version.inc flags 3 0x7f7fe6b5b6e1 2022-03-29T17:49:46.736 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method version.inc_conds flags 3 0x7f7fe6b5b6e1 2022-03-29T17:49:46.737 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method version.read flags 1 0x7f7fe6b5b21c 2022-03-29T17:49:46.737 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method version.check_conds flags 1 0x7f7fe6b5b4f9 2022-03-29T17:49:46.737 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - _load_class version success 2022-03-29T17:49:46.737 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - open_all_classes found rgw_gc 2022-03-29T17:49:46.737 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - _get_class adding new class name rgw_gc 0x612000029780 2022-03-29T17:49:46.738 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - _load_class rgw_gc from /usr/lib64/rados-classes/libcls_rgw_gc.so 2022-03-29T17:49:46.738 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,616 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw_gc/cls_rgw_gc.cc:539: Loaded rgw gc class! 2022-03-29T17:49:46.738 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_class rgw_gc status 3 2022-03-29T17:49:46.738 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_init flags 3 0x7f7fe691ea31 2022-03-29T17:49:46.739 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_enqueue flags 3 0x7f7fe691e130 2022-03-29T17:49:46.739 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_list_entries flags 1 0x7f7fe69202eb 2022-03-29T17:49:46.739 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_remove_entries flags 3 0x7f7fe691ed97 2022-03-29T17:49:46.739 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_update_entry flags 3 0x7f7fe6921e0e 2022-03-29T17:49:46.739 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:46,616 [shard 0] osd - _load_class rgw_gc success 2022-03-29T17:49:46.740 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:46,616 [shard 0] osd - OSD: nonce is 33978 2022-03-29T17:49:49.050 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:49,048 [shard 0] osd - _write_superblock writing superblock cluster_fsid 3be68405-f255-4011-9a3b-ba646329d493 osd_fsid 88a5bd02-6c7b-4069-9fdb-aea1c201b934 2022-03-29T17:49:49.050 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:49,048 [shard 0] osd - OSD::_write_superblock: do_transaction... 2022-03-29T17:49:49.055 INFO:teuthology.orchestra.run.smithi138.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 3be68405-f255-4011-9a3b-ba646329d493 2022-03-29T17:49:49.056 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:49,053 [shard 0] osd - stop 2022-03-29T17:49:49.056 INFO:teuthology.orchestra.run.smithi138.stderr:DEBUG 2022-03-29 17:49:49,053 [shard 0] osd - prepared to stop 2022-03-29T17:49:49.056 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:49,053 [shard 0] osd - stop 2022-03-29T17:49:49.056 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:49,053 [shard 0] alienstore - umount 2022-03-29T17:49:49.566 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:49,565 [shard 0] monc - stop 2022-03-29T17:49:49.567 INFO:teuthology.orchestra.run.smithi138.stderr:INFO 2022-03-29 17:49:49,565 [shard 0] mgrc - stop 2022-03-29T17:49:50.044 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-03-29T17:49:50.072 INFO:tasks.ceph:Reading keys from all nodes... 2022-03-29T17:49:50.072 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:50.072 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2022-03-29T17:49:50.138 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:50.139 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2022-03-29T17:49:50.204 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:50.205 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2022-03-29T17:49:50.270 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:50.271 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2022-03-29T17:49:50.336 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:50.337 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2022-03-29T17:49:50.393 INFO:tasks.ceph:Adding keys to all mons... 2022-03-29T17:49:50.394 DEBUG:teuthology.orchestra.run.smithi138:> sudo tee -a /etc/ceph/ceph.keyring 2022-03-29T17:49:50.458 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr.x] 2022-03-29T17:49:50.458 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQCuRkNiV+wMMRAAYEd3038LEvf6udlUcBmcWg== 2022-03-29T17:49:50.458 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.0] 2022-03-29T17:49:50.458 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQCxRkNil4+6NhAA1VJm184y0cEcWGz1bFYTVQ== 2022-03-29T17:49:50.459 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.1] 2022-03-29T17:49:50.459 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC2RkNiKQIVDhAA78hInYEMYiX5+Y/cksPrCQ== 2022-03-29T17:49:50.459 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.2] 2022-03-29T17:49:50.459 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC6RkNi/LvFJBAA63GcmXftpWdU4k3YX8Xj0w== 2022-03-29T17:49:50.460 INFO:teuthology.orchestra.run.smithi138.stdout:[client.0] 2022-03-29T17:49:50.460 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQCuRkNiJ6VGNBAAXbuupW/Y+UiGk3LaOU07Aw== 2022-03-29T17:49:50.461 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-03-29T17:49:50.563 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-03-29T17:49:50.625 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-03-29T17:49:50.687 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-03-29T17:49:50.749 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-03-29T17:49:50.814 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-03-29T17:49:50.815 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-03-29T17:49:50.841 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-03-29T17:49:50.955 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-03-29T17:49:50.981 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-03-29T17:49:51.038 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-03-29T17:49:51.038 INFO:tasks.ceph.mon.a:Restarting daemon 2022-03-29T17:49:51.039 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-03-29T17:49:51.081 INFO:tasks.ceph.mon.a:Started 2022-03-29T17:49:51.081 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-03-29T17:49:51.082 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-03-29T17:49:51.082 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-03-29T17:49:51.084 INFO:tasks.ceph.mgr.x:Started 2022-03-29T17:49:51.085 INFO:tasks.ceph:Setting crush tunables to default 2022-03-29T17:49:51.085 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd crush tunables default 2022-03-29T17:49:51.297 INFO:teuthology.orchestra.run.smithi138.stderr:adjusted tunables profile to default 2022-03-29T17:49:51.308 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-03-29T17:49:51.308 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:51.309 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2022-03-29T17:49:51.335 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:51.335 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2022-03-29T17:49:51.401 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:49:51.401 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2022-03-29T17:49:51.467 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 3e4be414-1d2c-4bc5-b6b2-d1f7948d012a 0 2022-03-29T17:49:51.477 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:51.475+0000 7fb027daaf40 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-03-29T17:49:51.696 INFO:teuthology.orchestra.run.smithi138.stdout:0 2022-03-29T17:49:51.710 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 59cc9d2e-e40e-4048-b5cb-31db1b464c16 1 2022-03-29T17:49:51.952 INFO:teuthology.orchestra.run.smithi138.stdout:1 2022-03-29T17:49:51.965 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 88a5bd02-6c7b-4069-9fdb-aea1c201b934 2 2022-03-29T17:49:52.000 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:51.998+0000 7fb027daaf40 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-03-29T17:49:52.182 INFO:teuthology.orchestra.run.smithi138.stdout:2 2022-03-29T17:49:52.194 INFO:tasks.ceph.osd.0:Restarting daemon 2022-03-29T17:49:52.195 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-03-29T17:49:52.237 INFO:tasks.ceph.osd.0:Started 2022-03-29T17:49:52.238 INFO:tasks.ceph.osd.1:Restarting daemon 2022-03-29T17:49:52.238 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-03-29T17:49:52.240 INFO:tasks.ceph.osd.1:Started 2022-03-29T17:49:52.241 INFO:tasks.ceph.osd.2:Restarting daemon 2022-03-29T17:49:52.241 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-03-29T17:49:52.243 INFO:tasks.ceph.osd.2:Started 2022-03-29T17:49:52.244 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-29T17:49:52.552 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:52.549+0000 7fb027daaf40 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-03-29T17:49:52.627 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:49:52.627 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":5,"fsid":"3be68405-f255-4011-9a3b-ba646329d493","created":"2022-03-29T17:49:51.193768+0000","modified":"2022-03-29T17:49:52.178337+0000","last_up_change":"0.000000","last_in_change":"2022-03-29T17:49:52.178337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"3e4be414-1d2c-4bc5-b6b2-d1f7948d012a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"59cc9d2e-e40e-4048-b5cb-31db1b464c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88a5bd02-6c7b-4069-9fdb-aea1c201b934","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-29T17:49:52.664 INFO:tasks.ceph.ceph_manager.ceph:[] 2022-03-29T17:49:52.665 INFO:tasks.ceph:Waiting for OSDs to come up 2022-03-29T17:49:52.825 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:52.822+0000 7fb027daaf40 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-03-29T17:49:52.967 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-03-29T17:49:53.000 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:52.997+0000 7fb027daaf40 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-03-29T17:49:53.092 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:53.090+0000 7fb027daaf40 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-03-29T17:49:53.196 INFO:tasks.ceph.osd.1.smithi138.stdout:warn: added seastar option --smp 1 2022-03-29T17:49:53.216 INFO:tasks.ceph.osd.1.smithi138.stderr:==35578==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2022-03-29T17:49:53.224 INFO:tasks.ceph.osd.1.smithi138.stderr:==35578==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd61583000; bottom 0x7fdee66be000; size: 0x001e7aec5000 (130911326208) 2022-03-29T17:49:53.224 INFO:tasks.ceph.osd.1.smithi138.stderr:False positive error reports may follow 2022-03-29T17:49:53.225 INFO:tasks.ceph.osd.1.smithi138.stderr:For details see https://github.com/google/sanitizers/issues/189 2022-03-29T17:49:53.231 INFO:tasks.ceph.osd.2.smithi138.stdout:warn: added seastar option --smp 1 2022-03-29T17:49:53.251 INFO:tasks.ceph.osd.2.smithi138.stderr:==35579==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2022-03-29T17:49:53.256 INFO:tasks.ceph.osd.2.smithi138.stderr:==35579==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff27530000; bottom 0x7f0d59c2b000; size: 0x00f1cd905000 (1038535905280) 2022-03-29T17:49:53.256 INFO:tasks.ceph.osd.2.smithi138.stderr:False positive error reports may follow 2022-03-29T17:49:53.257 INFO:tasks.ceph.osd.2.smithi138.stderr:For details see https://github.com/google/sanitizers/issues/189 2022-03-29T17:49:53.258 INFO:teuthology.misc.health.smithi138.stdout: 2022-03-29T17:49:53.259 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":5,"fsid":"3be68405-f255-4011-9a3b-ba646329d493","created":"2022-03-29T17:49:51.193768+0000","modified":"2022-03-29T17:49:52.178337+0000","last_up_change":"0.000000","last_in_change":"2022-03-29T17:49:52.178337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"3e4be414-1d2c-4bc5-b6b2-d1f7948d012a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"59cc9d2e-e40e-4048-b5cb-31db1b464c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88a5bd02-6c7b-4069-9fdb-aea1c201b934","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-29T17:49:53.271 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-03-29T17:49:53.276 INFO:tasks.ceph.osd.0.smithi138.stdout:warn: added seastar option --smp 1 2022-03-29T17:49:53.306 INFO:tasks.ceph.osd.1.smithi138.stdout:WARNING: debug mode. Not for benchmarking or production 2022-03-29T17:49:53.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,304 [shard 0] osd - open_all_classes 2022-03-29T17:49:53.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,305 [shard 0] osd - open_all_classes found journal 2022-03-29T17:49:53.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,305 [shard 0] osd - _get_class adding new class name journal 0x612000025d00 2022-03-29T17:49:53.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,305 [shard 0] osd - _load_class journal from /usr/lib64/rados-classes/libcls_journal.so 2022-03-29T17:49:53.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_class journal status 3 2022-03-29T17:49:53.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.create flags 3 0x7fdee51ed7de 2022-03-29T17:49:53.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.get_order flags 1 0x7fdee51ed1de 2022-03-29T17:49:53.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.get_splay_width flags 1 0x7fdee51ed246 2022-03-29T17:49:53.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.get_pool_id flags 1 0x7fdee51edc38 2022-03-29T17:49:53.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.get_minimum_set flags 1 0x7fdee51ede60 2022-03-29T17:49:53.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.set_minimum_set flags 3 0x7fdee51edfb6 2022-03-29T17:49:53.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.get_active_set flags 1 0x7fdee51eded2 2022-03-29T17:49:53.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.set_active_set flags 3 0x7fdee51ee155 2022-03-29T17:49:53.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.get_client flags 1 0x7fdee51ef4e4 2022-03-29T17:49:53.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.client_register flags 3 0x7fdee51f0c5e 2022-03-29T17:49:53.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.client_update_data flags 3 0x7fdee51ef7bf 2022-03-29T17:49:53.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.client_update_state flags 3 0x7fdee51efc9f 2022-03-29T17:49:53.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.client_unregister flags 3 0x7fdee51f2e8a 2022-03-29T17:49:53.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.client_commit flags 3 0x7fdee51f3832 2022-03-29T17:49:53.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.client_list flags 1 0x7fdee51f130f 2022-03-29T17:49:53.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.get_next_tag_tid flags 1 0x7fdee51edf44 2022-03-29T17:49:53.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.get_tag flags 1 0x7fdee51eef5d 2022-03-29T17:49:53.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.tag_create flags 3 0x7fdee51f3129 2022-03-29T17:49:53.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.tag_list flags 1 0x7fdee51f1573 2022-03-29T17:49:53.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.guard_append flags 3 0x7fdee51ecb5c 2022-03-29T17:49:53.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - register_cxx_method journal.append flags 3 0x7fdee51ee2f4 2022-03-29T17:49:53.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - _load_class journal success 2022-03-29T17:49:53.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - open_all_classes found rgw 2022-03-29T17:49:53.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - _get_class adding new class name rgw 0x612000025e80 2022-03-29T17:49:53.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,306 [shard 0] osd - _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so 2022-03-29T17:49:53.344 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,310 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw/cls_rgw.cc:4353: Loaded rgw class! 2022-03-29T17:49:53.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_class rgw status 3 2022-03-29T17:49:53.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_cxx_method rgw.bucket_init_index flags 3 0x7fdee4ea2175 2022-03-29T17:49:53.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7fdee4eb1ca1 2022-03-29T17:49:53.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_cxx_method rgw.bucket_list flags 1 0x7fdee4ebe331 2022-03-29T17:49:53.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_cxx_method rgw.bucket_check_index flags 1 0x7fdee4ebcea4 2022-03-29T17:49:53.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_cxx_method rgw.bucket_rebuild_index flags 3 0x7fdee4ebd70f 2022-03-29T17:49:53.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_cxx_method rgw.bucket_update_stats flags 3 0x7fdee4eb35bd 2022-03-29T17:49:53.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_cxx_method rgw.bucket_prepare_op flags 3 0x7fdee4ead3dd 2022-03-29T17:49:53.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,310 [shard 0] osd - register_cxx_method rgw.bucket_complete_op flags 3 0x7fdee4eb525b 2022-03-29T17:49:53.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bucket_link_olh flags 3 0x7fdee4ec049b 2022-03-29T17:49:53.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bucket_unlink_instance flags 3 0x7fdee4eb7434 2022-03-29T17:49:53.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bucket_read_olh_log flags 1 0x7fdee4eb8d50 2022-03-29T17:49:53.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7fdee4eb6dcb 2022-03-29T17:49:53.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bucket_clear_olh flags 3 0x7fdee4eb661e 2022-03-29T17:49:53.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.obj_remove flags 3 0x7fdee4eba48a 2022-03-29T17:49:53.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.obj_store_pg_ver flags 2 0x7fdee4ea6019 2022-03-29T17:49:53.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7fdee4ea5cda 2022-03-29T17:49:53.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.obj_check_mtime flags 1 0x7fdee4ea5a18 2022-03-29T17:49:53.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bi_get flags 1 0x7fdee4ea54fe 2022-03-29T17:49:53.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bi_put flags 3 0x7fdee4ea5304 2022-03-29T17:49:53.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bi_list flags 1 0x7fdee4ebbb1e 2022-03-29T17:49:53.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bi_log_list flags 1 0x7fdee4eacf1a 2022-03-29T17:49:53.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bi_log_trim flags 3 0x7fdee4ea4d56 2022-03-29T17:49:53.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.dir_suggest_changes flags 3 0x7fdee4eb393a 2022-03-29T17:49:53.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bi_log_resync flags 3 0x7fdee4eb2fe3 2022-03-29T17:49:53.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.bi_log_stop flags 3 0x7fdee4eb2a09 2022-03-29T17:49:53.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.user_usage_log_add flags 3 0x7fdee4eaee48 2022-03-29T17:49:53.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.user_usage_log_read flags 1 0x7fdee4eae6db 2022-03-29T17:49:53.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.user_usage_log_trim flags 3 0x7fdee4eaebf9 2022-03-29T17:49:53.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.usage_log_clear flags 2 0x7fdee4e9fdda 2022-03-29T17:49:53.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.gc_set_entry flags 3 0x7fdee4ea838e 2022-03-29T17:49:53.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.gc_defer_entry flags 3 0x7fdee4ea8244 2022-03-29T17:49:53.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.gc_list flags 1 0x7fdee4ea69c9 2022-03-29T17:49:53.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.gc_remove flags 3 0x7fdee4ebdeb9 2022-03-29T17:49:53.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.lc_get_entry flags 1 0x7fdee4ea7423 2022-03-29T17:49:53.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.lc_set_entry flags 3 0x7fdee4ea9271 2022-03-29T17:49:53.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.lc_rm_entry flags 3 0x7fdee4ea9122 2022-03-29T17:49:53.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.lc_get_next_entry flags 1 0x7fdee4ea4850 2022-03-29T17:49:53.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.lc_put_head flags 3 0x7fdee4ea45fb 2022-03-29T17:49:53.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.lc_get_head flags 1 0x7fdee4ea42c1 2022-03-29T17:49:53.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.lc_list_entries flags 1 0x7fdee4eab3ee 2022-03-29T17:49:53.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.reshard_add flags 3 0x7fdee4ea3cbc 2022-03-29T17:49:53.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.reshard_list flags 1 0x7fdee4ea8941 2022-03-29T17:49:53.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.reshard_get flags 1 0x7fdee4ea35af 2022-03-29T17:49:53.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.reshard_remove flags 3 0x7fdee4ea3188 2022-03-29T17:49:53.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.set_bucket_resharding flags 3 0x7fdee4eb274a 2022-03-29T17:49:53.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.clear_bucket_resharding flags 3 0x7fdee4eb24ee 2022-03-29T17:49:53.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.guard_bucket_resharding flags 1 0x7fdee4eb2291 2022-03-29T17:49:53.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - register_cxx_method rgw.get_bucket_resharding flags 1 0x7fdee4eb1eea 2022-03-29T17:49:53.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - _load_class rgw success 2022-03-29T17:49:53.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - open_all_classes found cephfs 2022-03-29T17:49:53.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - _get_class adding new class name cephfs 0x612000026180 2022-03-29T17:49:53.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,311 [shard 0] osd - _load_class cephfs from /usr/lib64/rados-classes/libcls_cephfs.so 2022-03-29T17:49:53.357 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,312 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cephfs/cls_cephfs.cc:201: loading cephfs 2022-03-29T17:49:53.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,312 [shard 0] osd - register_class cephfs status 3 2022-03-29T17:49:53.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,312 [shard 0] osd - register_cxx_method cephfs.accumulate_inode_metadata flags 3 0x7fdee4bed2f9 2022-03-29T17:49:53.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,312 [shard 0] osd - _load_class cephfs success 2022-03-29T17:49:53.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,312 [shard 0] osd - open_all_classes found numops 2022-03-29T17:49:53.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,312 [shard 0] osd - _get_class adding new class name numops 0x612000026480 2022-03-29T17:49:53.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,312 [shard 0] osd - _load_class numops from /usr/lib64/rados-classes/libcls_numops.so 2022-03-29T17:49:53.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,312 [shard 0] osd - register_class numops status 3 2022-03-29T17:49:53.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method numops.add flags 3 0x7fdee49df65f 2022-03-29T17:49:53.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method numops.mul flags 3 0x7fdee49debe6 2022-03-29T17:49:53.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - _load_class numops success 2022-03-29T17:49:53.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - open_all_classes found lua 2022-03-29T17:49:53.359 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,313 [shard 0] osd - _get_class not permitted to load lua 2022-03-29T17:49:53.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - open_all_classes found test_remote_reads 2022-03-29T17:49:53.360 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,313 [shard 0] osd - _get_class not permitted to load test_remote_reads 2022-03-29T17:49:53.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - open_all_classes found rbd 2022-03-29T17:49:53.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - _get_class adding new class name rbd 0x612000026780 2022-03-29T17:49:53.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so 2022-03-29T17:49:53.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_class rbd status 3 2022-03-29T17:49:53.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.create flags 3 0x7fdee477fec0 2022-03-29T17:49:53.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_features flags 1 0x7fdee476d5af 2022-03-29T17:49:53.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.set_features flags 3 0x7fdee476d7d2 2022-03-29T17:49:53.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_size flags 1 0x7fdee4792cb9 2022-03-29T17:49:53.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.set_size flags 3 0x7fdee477275d 2022-03-29T17:49:53.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_snapcontext flags 1 0x7fdee476e27a 2022-03-29T17:49:53.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_object_prefix flags 1 0x7fdee476caaf 2022-03-29T17:49:53.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_data_pool flags 1 0x7fdee476c5be 2022-03-29T17:49:53.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_snapshot_name flags 1 0x7fdee478bf69 2022-03-29T17:49:53.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_snapshot_timestamp flags 1 0x7fdee478bbc7 2022-03-29T17:49:53.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.snapshot_get flags 1 0x7fdee4790dcd 2022-03-29T17:49:53.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.snapshot_add flags 3 0x7fdee47979a5 2022-03-29T17:49:53.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.snapshot_remove flags 3 0x7fdee47900e4 2022-03-29T17:49:53.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.snapshot_rename flags 3 0x7fdee478d938 2022-03-29T17:49:53.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.snapshot_trash_add flags 3 0x7fdee478c82f 2022-03-29T17:49:53.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_all_features flags 1 0x7fdee47630bc 2022-03-29T17:49:53.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_parent flags 1 0x7fdee478ad9f 2022-03-29T17:49:53.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.set_parent flags 3 0x7fdee477d007 2022-03-29T17:49:53.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.remove_parent flags 3 0x7fdee478fa52 2022-03-29T17:49:53.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.parent_get flags 1 0x7fdee478b91f 2022-03-29T17:49:53.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.parent_overlap_get flags 1 0x7fdee4793270 2022-03-29T17:49:53.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.parent_attach flags 3 0x7fdee4773489 2022-03-29T17:49:53.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.parent_detach flags 3 0x7fdee478fa72 2022-03-29T17:49:53.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.set_protection_status flags 3 0x7fdee47927f9 2022-03-29T17:49:53.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,313 [shard 0] osd - register_cxx_method rbd.get_protection_status flags 1 0x7fdee4791655 2022-03-29T17:49:53.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.get_stripe_unit_count flags 1 0x7fdee4770140 2022-03-29T17:49:53.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.set_stripe_unit_count flags 3 0x7fdee476f5c9 2022-03-29T17:49:53.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.get_create_timestamp flags 1 0x7fdee476ce65 2022-03-29T17:49:53.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.get_access_timestamp flags 1 0x7fdee476d0d3 2022-03-29T17:49:53.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.get_modify_timestamp flags 1 0x7fdee476d341 2022-03-29T17:49:53.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.get_flags flags 1 0x7fdee4793842 2022-03-29T17:49:53.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.set_flags flags 3 0x7fdee478fa92 2022-03-29T17:49:53.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.op_features_get flags 1 0x7fdee476c44f 2022-03-29T17:49:53.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.op_features_set flags 3 0x7fdee4770a28 2022-03-29T17:49:53.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.metadata_list flags 1 0x7fdee4781d62 2022-03-29T17:49:53.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.metadata_set flags 3 0x7fdee478a5a7 2022-03-29T17:49:53.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.metadata_remove flags 3 0x7fdee4769e87 2022-03-29T17:49:53.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.metadata_get flags 1 0x7fdee476a065 2022-03-29T17:49:53.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.snapshot_get_limit flags 1 0x7fdee476c731 2022-03-29T17:49:53.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.snapshot_set_limit flags 3 0x7fdee476f079 2022-03-29T17:49:53.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.child_attach flags 3 0x7fdee479073a 2022-03-29T17:49:53.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.child_detach flags 3 0x7fdee47920fc 2022-03-29T17:49:53.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.children_list flags 1 0x7fdee478c329 2022-03-29T17:49:53.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.migration_set flags 3 0x7fdee4771b72 2022-03-29T17:49:53.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.migration_set_state flags 3 0x7fdee4771dc9 2022-03-29T17:49:53.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.migration_get flags 1 0x7fdee476e075 2022-03-29T17:49:53.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.migration_remove flags 3 0x7fdee4770125 2022-03-29T17:49:53.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.set_modify_timestamp flags 3 0x7fdee476c9aa 2022-03-29T17:49:53.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.set_access_timestamp flags 3 0x7fdee476c8a5 2022-03-29T17:49:53.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.add_child flags 3 0x7fdee4794af5 2022-03-29T17:49:53.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.remove_child flags 3 0x7fdee4795052 2022-03-29T17:49:53.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.get_children flags 1 0x7fdee4795633 2022-03-29T17:49:53.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.get_id flags 1 0x7fdee476548d 2022-03-29T17:49:53.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.set_id flags 3 0x7fdee47656f0 2022-03-29T17:49:53.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.dir_get_id flags 1 0x7fdee4768c9d 2022-03-29T17:49:53.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.dir_get_name flags 1 0x7fdee476959c 2022-03-29T17:49:53.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.dir_list flags 1 0x7fdee477f5f8 2022-03-29T17:49:53.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.dir_add_image flags 3 0x7fdee4783a41 2022-03-29T17:49:53.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.dir_remove_image flags 3 0x7fdee476941b 2022-03-29T17:49:53.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.dir_rename_image flags 3 0x7fdee478380d 2022-03-29T17:49:53.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.dir_state_assert flags 1 0x7fdee476c33d 2022-03-29T17:49:53.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.dir_state_set flags 3 0x7fdee476edf0 2022-03-29T17:49:53.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.object_map_load flags 1 0x7fdee4783dee 2022-03-29T17:49:53.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.object_map_save flags 3 0x7fdee47842fc 2022-03-29T17:49:53.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.object_map_resize flags 3 0x7fdee4785a9b 2022-03-29T17:49:53.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.object_map_update flags 3 0x7fdee4784bce 2022-03-29T17:49:53.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.object_map_snap_add flags 3 0x7fdee4783ec6 2022-03-29T17:49:53.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.object_map_snap_remove flags 3 0x7fdee47844bf 2022-03-29T17:49:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.snap_list flags 1 0x7fdee476bf91 2022-03-29T17:49:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.snap_add flags 3 0x7fdee476e68e 2022-03-29T17:49:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.snap_remove flags 3 0x7fdee4765b41 2022-03-29T17:49:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.snap_rename flags 3 0x7fdee47662a2 2022-03-29T17:49:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.mirror_uuid_get flags 1 0x7fdee476bcef 2022-03-29T17:49:53.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.mirror_uuid_set flags 3 0x7fdee4778253 2022-03-29T17:49:53.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.mirror_mode_get flags 1 0x7fdee4766c4b 2022-03-29T17:49:53.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.mirror_mode_set flags 3 0x7fdee477bba6 2022-03-29T17:49:53.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,314 [shard 0] osd - register_cxx_method rbd.mirror_peer_ping flags 3 0x7fdee478a3fb 2022-03-29T17:49:53.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_peer_list flags 1 0x7fdee477c035 2022-03-29T17:49:53.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_peer_add flags 3 0x7fdee477d2de 2022-03-29T17:49:53.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_peer_remove flags 3 0x7fdee477413d 2022-03-29T17:49:53.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_client flags 3 0x7fdee4773b04 2022-03-29T17:49:53.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_cluster flags 3 0x7fdee477c18f 2022-03-29T17:49:53.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_direction flags 3 0x7fdee4773ddb 2022-03-29T17:49:53.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_list flags 1 0x7fdee477ecae 2022-03-29T17:49:53.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_get_image_id flags 1 0x7fdee4774e8c 2022-03-29T17:49:53.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_get flags 1 0x7fdee477449a 2022-03-29T17:49:53.381 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_set flags 3 0x7fdee4774d0f 2022-03-29T17:49:53.381 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_remove flags 3 0x7fdee477af25 2022-03-29T17:49:53.381 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_status_set flags 11 0x7fdee477a7da 2022-03-29T17:49:53.381 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove flags 3 0x7fdee477b02f 2022-03-29T17:49:53.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get flags 1 0x7fdee477ce5f 2022-03-29T17:49:53.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_status_list flags 1 0x7fdee4786d7f 2022-03-29T17:49:53.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get_summary flags 1 0x7fdee4787570 2022-03-29T17:49:53.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove_down flags 3 0x7fdee4779f42 2022-03-29T17:49:53.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_get flags 1 0x7fdee477a2cd 2022-03-29T17:49:53.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_list flags 1 0x7fdee478804d 2022-03-29T17:49:53.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_instances_list flags 1 0x7fdee4778851 2022-03-29T17:49:53.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_instances_add flags 11 0x7fdee47754de 2022-03-29T17:49:53.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_instances_remove flags 3 0x7fdee47756ef 2022-03-29T17:49:53.384 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_map_list flags 1 0x7fdee477805e 2022-03-29T17:49:53.384 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_map_update flags 2 0x7fdee4775841 2022-03-29T17:49:53.384 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_map_remove flags 2 0x7fdee4775bbd 2022-03-29T17:49:53.384 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_unlink_peer flags 3 0x7fdee4796b3c 2022-03-29T17:49:53.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_set_copy_progress flags 3 0x7fdee4795e49 2022-03-29T17:49:53.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.group_dir_list flags 1 0x7fdee477e196 2022-03-29T17:49:53.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.group_dir_add flags 3 0x7fdee4782ca7 2022-03-29T17:49:53.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.group_dir_remove flags 3 0x7fdee4769cc2 2022-03-29T17:49:53.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.group_dir_rename flags 3 0x7fdee4782ec2 2022-03-29T17:49:53.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.group_image_remove flags 3 0x7fdee476332a 2022-03-29T17:49:53.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.group_image_list flags 1 0x7fdee4778c9a 2022-03-29T17:49:53.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.group_image_set flags 3 0x7fdee4766cb3 2022-03-29T17:49:53.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.image_group_add flags 3 0x7fdee4770b03 2022-03-29T17:49:53.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,315 [shard 0] osd - register_cxx_method rbd.image_group_remove flags 3 0x7fdee4770f46 2022-03-29T17:49:53.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.image_group_get flags 1 0x7fdee476cc65 2022-03-29T17:49:53.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.group_snap_set flags 3 0x7fdee4788ae6 2022-03-29T17:49:53.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.group_snap_remove flags 3 0x7fdee476ab76 2022-03-29T17:49:53.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.group_snap_get_by_id flags 1 0x7fdee476ad09 2022-03-29T17:49:53.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.group_snap_list flags 1 0x7fdee477dc44 2022-03-29T17:49:53.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.trash_add flags 3 0x7fdee478179b 2022-03-29T17:49:53.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.trash_remove flags 3 0x7fdee4775de3 2022-03-29T17:49:53.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.trash_list flags 1 0x7fdee4788ece 2022-03-29T17:49:53.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.trash_get flags 1 0x7fdee47760e8 2022-03-29T17:49:53.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.trash_state_set flags 3 0x7fdee477636e 2022-03-29T17:49:53.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.namespace_add flags 3 0x7fdee47767a6 2022-03-29T17:49:53.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.namespace_remove flags 3 0x7fdee47769d1 2022-03-29T17:49:53.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.namespace_list flags 1 0x7fdee4776bba 2022-03-29T17:49:53.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.copyup flags 3 0x7fdee476316b 2022-03-29T17:49:53.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.sparse_copyup flags 3 0x7fdee4793e66 2022-03-29T17:49:53.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.assert_snapc_seq flags 3 0x7fdee476321b 2022-03-29T17:49:53.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - register_cxx_method rbd.sparsify flags 3 0x7fdee4767074 2022-03-29T17:49:53.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - _load_class rbd success 2022-03-29T17:49:53.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - open_all_classes found queue 2022-03-29T17:49:53.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - _get_class adding new class name queue 0x612000026900 2022-03-29T17:49:53.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,316 [shard 0] osd - _load_class queue from /usr/lib64/rados-classes/libcls_queue.so 2022-03-29T17:49:53.392 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,317 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/queue/cls_queue.cc:125: Loaded queue class! 2022-03-29T17:49:53.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_class queue status 3 2022-03-29T17:49:53.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method queue.queue_init flags 3 0x7fdee4527256 2022-03-29T17:49:53.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method queue.queue_get_capacity flags 1 0x7fdee45269e9 2022-03-29T17:49:53.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method queue.queue_enqueue flags 3 0x7fdee452752e 2022-03-29T17:49:53.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method queue.queue_list_entries flags 1 0x7fdee4526daf 2022-03-29T17:49:53.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method queue.queue_remove_entries flags 3 0x7fdee4526b81 2022-03-29T17:49:53.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - _load_class queue success 2022-03-29T17:49:53.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - open_all_classes found user 2022-03-29T17:49:53.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - _get_class adding new class name user 0x612000026c00 2022-03-29T17:49:53.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - _load_class user from /usr/lib64/rados-classes/libcls_user.so 2022-03-29T17:49:53.394 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,317 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/user/cls_user.cc:506: Loaded user class! 2022-03-29T17:49:53.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_class user status 3 2022-03-29T17:49:53.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method user.set_buckets_info flags 3 0x7fdee430dc4b 2022-03-29T17:49:53.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method user.complete_stats_sync flags 3 0x7fdee430d9a3 2022-03-29T17:49:53.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method user.remove_bucket flags 3 0x7fdee430d1be 2022-03-29T17:49:53.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method user.list_buckets flags 1 0x7fdee430e733 2022-03-29T17:49:53.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method user.get_header flags 1 0x7fdee430cfc7 2022-03-29T17:49:53.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method user.reset_user_stats flags 3 0x7fdee430c54c 2022-03-29T17:49:53.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - register_cxx_method user.reset_user_stats2 flags 3 0x7fdee430f19b 2022-03-29T17:49:53.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - _load_class user success 2022-03-29T17:49:53.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - open_all_classes found cas 2022-03-29T17:49:53.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - _get_class adding new class name cas 0x612000027080 2022-03-29T17:49:53.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,317 [shard 0] osd - _load_class cas from /usr/lib64/rados-classes/libcls_cas.so 2022-03-29T17:49:53.397 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,318 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cas/cls_cas.cc:211: Loaded cas class! 2022-03-29T17:49:53.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - register_class cas status 3 2022-03-29T17:49:53.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - register_cxx_method cas.chunk_create_or_get_ref flags 3 0x7fdee40f6b38 2022-03-29T17:49:53.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - register_cxx_method cas.chunk_get_ref flags 3 0x7fdee40f6848 2022-03-29T17:49:53.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - register_cxx_method cas.chunk_put_ref flags 3 0x7fdee40f64c4 2022-03-29T17:49:53.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - register_cxx_method cas.references_chunk flags 1 0x7fdee40f61fc 2022-03-29T17:49:53.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - _load_class cas success 2022-03-29T17:49:53.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - open_all_classes found fifo 2022-03-29T17:49:53.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - _get_class adding new class name fifo 0x612000027380 2022-03-29T17:49:53.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - _load_class fifo from /usr/lib64/rados-classes/libcls_fifo.so 2022-03-29T17:49:53.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - register_class fifo status 3 2022-03-29T17:49:53.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,318 [shard 0] osd - register_cxx_method fifo.create_meta flags 3 0x7fdee3eb7dee 2022-03-29T17:49:53.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method fifo.get_meta flags 1 0x7fdee3eb90b2 2022-03-29T17:49:53.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method fifo.update_meta flags 3 0x7fdee3eb957b 2022-03-29T17:49:53.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method fifo.init_part flags 3 0x7fdee3eb6cf2 2022-03-29T17:49:53.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method fifo.push_part flags 3 0x7fdee3eb73e5 2022-03-29T17:49:53.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method fifo.trim_part flags 3 0x7fdee3eb6782 2022-03-29T17:49:53.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method fifo.part_list flags 1 0x7fdee3eb5ed9 2022-03-29T17:49:53.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method fifo.get_part_info flags 1 0x7fdee3eb577e 2022-03-29T17:49:53.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - _load_class fifo success 2022-03-29T17:49:53.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - open_all_classes found cmpomap 2022-03-29T17:49:53.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - _get_class adding new class name cmpomap 0x612000027680 2022-03-29T17:49:53.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - _load_class cmpomap from /usr/lib64/rados-classes/libcls_cmpomap.so 2022-03-29T17:49:53.402 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,319 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cmpomap/server.cc:287: Loaded cmpomap class! 2022-03-29T17:49:53.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_class cmpomap status 3 2022-03-29T17:49:53.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method cmpomap.cmp_vals flags 1 0x7fdee3c9a2e7 2022-03-29T17:49:53.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method cmpomap.cmp_set_vals flags 3 0x7fdee3c9970e 2022-03-29T17:49:53.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - register_cxx_method cmpomap.cmp_rm_keys flags 3 0x7fdee3c99136 2022-03-29T17:49:53.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - _load_class cmpomap success 2022-03-29T17:49:53.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - open_all_classes found hello 2022-03-29T17:49:53.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - _get_class adding new class name hello 0x612000027b00 2022-03-29T17:49:53.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,319 [shard 0] osd - _load_class hello from /usr/lib64/rados-classes/libcls_hello.so 2022-03-29T17:49:53.404 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,320 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/hello/cls_hello.cc:316: loading cls_hello 2022-03-29T17:49:53.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_class hello status 3 2022-03-29T17:49:53.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.say_hello flags 1 0x7fdee3a89bf6 2022-03-29T17:49:53.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.record_hello flags 10 0x7fdee3a89ea9 2022-03-29T17:49:53.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.write_return_data flags 2 0x7fdee3a89d7f 2022-03-29T17:49:53.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.writes_dont_return_data flags 2 0x7fdee3a89d7f 2022-03-29T17:49:53.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.write_too_much_return_data flags 2 0x7fdee3a89c6d 2022-03-29T17:49:53.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.replay flags 1 0x7fdee3a89bce 2022-03-29T17:49:53.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.turn_it_to_11 flags 11 0x7fdee3a8a808 2022-03-29T17:49:53.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.bad_reader flags 2 0x7fdee3a89baf 2022-03-29T17:49:53.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method hello.bad_writer flags 1 0x7fdee3a89b9d 2022-03-29T17:49:53.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - _load_class hello success 2022-03-29T17:49:53.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - open_all_classes found log 2022-03-29T17:49:53.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - _get_class adding new class name log 0x612000027e00 2022-03-29T17:49:53.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - _load_class log from /usr/lib64/rados-classes/libcls_log.so 2022-03-29T17:49:53.408 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,320 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/log/cls_log.cc:305: Loaded log class! 2022-03-29T17:49:53.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_class log status 3 2022-03-29T17:49:53.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method log.add flags 3 0x7fdee3878f1f 2022-03-29T17:49:53.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method log.list flags 1 0x7fdee3879a5d 2022-03-29T17:49:53.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method log.trim flags 3 0x7fdee38792cf 2022-03-29T17:49:53.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - register_cxx_method log.info flags 1 0x7fdee3878af7 2022-03-29T17:49:53.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - _load_class log success 2022-03-29T17:49:53.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - open_all_classes found otp 2022-03-29T17:49:53.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,320 [shard 0] osd - _get_class adding new class name otp 0x612000028280 2022-03-29T17:49:53.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - _load_class otp from /usr/lib64/rados-classes/libcls_otp.so 2022-03-29T17:49:53.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - register_class otp status 3 2022-03-29T17:49:53.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - register_cxx_method otp.otp_set flags 3 0x7fdee366537f 2022-03-29T17:49:53.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - register_cxx_method otp.otp_get flags 1 0x7fdee366498f 2022-03-29T17:49:53.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - register_cxx_method otp.otp_check flags 3 0x7fdee3663fc3 2022-03-29T17:49:53.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - register_cxx_method otp.otp_get_result flags 1 0x7fdee3663902 2022-03-29T17:49:53.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - register_cxx_method otp.otp_remove flags 3 0x7fdee3664700 2022-03-29T17:49:53.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - register_cxx_method otp.get_current_time flags 1 0x7fdee36635f0 2022-03-29T17:49:53.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - _load_class otp success 2022-03-29T17:49:53.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - open_all_classes found sdk 2022-03-29T17:49:53.412 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,321 [shard 0] osd - _get_class not permitted to load sdk 2022-03-29T17:49:53.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - open_all_classes found timeindex 2022-03-29T17:49:53.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - _get_class adding new class name timeindex 0x612000028400 2022-03-29T17:49:53.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,321 [shard 0] osd - _load_class timeindex from /usr/lib64/rados-classes/libcls_timeindex.so 2022-03-29T17:49:53.413 INFO:tasks.ceph.osd.0.smithi138.stderr:==35577==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2022-03-29T17:49:53.414 INFO:tasks.ceph.osd.0.smithi138.stderr:==35577==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc12759000; bottom 0x7fb601382000; size: 0x0046113d7000 (300936949760) 2022-03-29T17:49:53.414 INFO:tasks.ceph.osd.0.smithi138.stderr:False positive error reports may follow 2022-03-29T17:49:53.414 INFO:tasks.ceph.osd.0.smithi138.stderr:For details see https://github.com/google/sanitizers/issues/189 2022-03-29T17:49:53.423 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,338 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/timeindex/cls_timeindex.cc:247: Loaded timeindex class! 2022-03-29T17:49:53.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,338 [shard 0] osd - register_class timeindex status 3 2022-03-29T17:49:53.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,338 [shard 0] osd - register_cxx_method timeindex.add flags 3 0x7fdee3236894 2022-03-29T17:49:53.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,338 [shard 0] osd - register_cxx_method timeindex.list flags 1 0x7fdee3236c23 2022-03-29T17:49:53.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,338 [shard 0] osd - register_cxx_method timeindex.trim flags 3 0x7fdee323643d 2022-03-29T17:49:53.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,338 [shard 0] osd - _load_class timeindex success 2022-03-29T17:49:53.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,338 [shard 0] osd - open_all_classes found lock 2022-03-29T17:49:53.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,338 [shard 0] osd - _get_class adding new class name lock 0x612000028880 2022-03-29T17:49:53.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,339 [shard 0] osd - _load_class lock from /usr/lib64/rados-classes/libcls_lock.so 2022-03-29T17:49:53.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,339 [shard 0] osd - register_class lock status 3 2022-03-29T17:49:53.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,339 [shard 0] osd - register_cxx_method lock.lock flags 11 0x7fdee30234af 2022-03-29T17:49:53.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,339 [shard 0] osd - register_cxx_method lock.unlock flags 11 0x7fdee3020e27 2022-03-29T17:49:53.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,339 [shard 0] osd - register_cxx_method lock.break_lock flags 3 0x7fdee3020ccd 2022-03-29T17:49:53.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,339 [shard 0] osd - register_cxx_method lock.get_info flags 1 0x7fdee3021a33 2022-03-29T17:49:53.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,340 [shard 0] osd - register_cxx_method lock.list_locks flags 1 0x7fdee301f26b 2022-03-29T17:49:53.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,340 [shard 0] osd - register_cxx_method lock.assert_locked flags 9 0x7fdee30203bd 2022-03-29T17:49:53.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,340 [shard 0] osd - register_cxx_method lock.set_cookie flags 11 0x7fdee3020ff4 2022-03-29T17:49:53.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,340 [shard 0] osd - _load_class lock success 2022-03-29T17:49:53.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,340 [shard 0] osd - open_all_classes found 2pc_queue 2022-03-29T17:49:53.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,340 [shard 0] osd - _get_class adding new class name 2pc_queue 0x612000028a00 2022-03-29T17:49:53.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,340 [shard 0] osd - _load_class 2pc_queue from /usr/lib64/rados-classes/libcls_2pc_queue.so 2022-03-29T17:49:53.427 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,341 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/2pc_queue/cls_2pc_queue.cc:575: Loaded 2pc queue class! 2022-03-29T17:49:53.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_class 2pc_queue status 3 2022-03-29T17:49:53.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_init flags 3 0x7fdee2de26a3 2022-03-29T17:49:53.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_get_capacity flags 1 0x7fdee2de1c19 2022-03-29T17:49:53.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_reserve flags 3 0x7fdee2de5504 2022-03-29T17:49:53.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_commit flags 3 0x7fdee2de2b0d 2022-03-29T17:49:53.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_abort flags 3 0x7fdee2de428f 2022-03-29T17:49:53.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_reservations flags 1 0x7fdee2de4c8c 2022-03-29T17:49:53.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_entries flags 1 0x7fdee2de2076 2022-03-29T17:49:53.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_remove_entries flags 3 0x7fdee2de1e39 2022-03-29T17:49:53.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_expire_reservations flags 3 0x7fdee2de37d0 2022-03-29T17:49:53.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - _load_class 2pc_queue success 2022-03-29T17:49:53.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - open_all_classes found refcount 2022-03-29T17:49:53.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - _get_class adding new class name refcount 0x612000028d00 2022-03-29T17:49:53.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,341 [shard 0] osd - _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so 2022-03-29T17:49:53.431 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,346 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/refcount/cls_refcount.cc:199: Loaded refcount class! 2022-03-29T17:49:53.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - register_class refcount status 3 2022-03-29T17:49:53.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - register_cxx_method refcount.get flags 3 0x7fdee2b1ac89 2022-03-29T17:49:53.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - register_cxx_method refcount.put flags 3 0x7fdee2b1a94c 2022-03-29T17:49:53.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - register_cxx_method refcount.set flags 3 0x7fdee2b1a0c1 2022-03-29T17:49:53.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - register_cxx_method refcount.read flags 1 0x7fdee2b1a602 2022-03-29T17:49:53.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - _load_class refcount success 2022-03-29T17:49:53.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - open_all_classes found version 2022-03-29T17:49:53.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - _get_class adding new class name version 0x612000029000 2022-03-29T17:49:53.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,346 [shard 0] osd - _load_class version from /usr/lib64/rados-classes/libcls_version.so 2022-03-29T17:49:53.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,347 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/version/cls_version.cc:218: Loaded version class! 2022-03-29T17:49:53.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - register_class version status 3 2022-03-29T17:49:53.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - register_cxx_method version.set flags 3 0x7fdee28693d7 2022-03-29T17:49:53.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - register_cxx_method version.inc flags 3 0x7fdee28696e1 2022-03-29T17:49:53.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - register_cxx_method version.inc_conds flags 3 0x7fdee28696e1 2022-03-29T17:49:53.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - register_cxx_method version.read flags 1 0x7fdee286921c 2022-03-29T17:49:53.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - register_cxx_method version.check_conds flags 1 0x7fdee28694f9 2022-03-29T17:49:53.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - _load_class version success 2022-03-29T17:49:53.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - open_all_classes found rgw_gc 2022-03-29T17:49:53.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - _get_class adding new class name rgw_gc 0x612000029480 2022-03-29T17:49:53.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,347 [shard 0] osd - _load_class rgw_gc from /usr/lib64/rados-classes/libcls_rgw_gc.so 2022-03-29T17:49:53.436 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,347 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw_gc/cls_rgw_gc.cc:539: Loaded rgw gc class! 2022-03-29T17:49:53.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,348 [shard 0] osd - register_class rgw_gc status 3 2022-03-29T17:49:53.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,348 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_init flags 3 0x7fdee262ca31 2022-03-29T17:49:53.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,348 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_enqueue flags 3 0x7fdee262c130 2022-03-29T17:49:53.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,348 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_list_entries flags 1 0x7fdee262e2eb 2022-03-29T17:49:53.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,348 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_remove_entries flags 3 0x7fdee262cd97 2022-03-29T17:49:53.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,348 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_update_entry flags 3 0x7fdee262fe0e 2022-03-29T17:49:53.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,348 [shard 0] osd - _load_class rgw_gc success 2022-03-29T17:49:53.437 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,348 [shard 0] osd - OSD: nonce is 35578 2022-03-29T17:49:53.438 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding auth protocol: cephx 2022-03-29T17:49:53.438 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding auth protocol: cephx 2022-03-29T17:49:53.438 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding auth protocol: cephx 2022-03-29T17:49:53.438 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding con mode: crc 2022-03-29T17:49:53.439 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding con mode: crc 2022-03-29T17:49:53.439 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding con mode: crc 2022-03-29T17:49:53.439 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding con mode: crc 2022-03-29T17:49:53.439 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding con mode: crc 2022-03-29T17:49:53.439 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,349 [shard 0] auth - AuthRegistry(0x7fdee261eb60) adding con mode: crc 2022-03-29T17:49:53.440 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,350 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-1/keyring 2022-03-29T17:49:53.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,353 [shard 0] monc - build_monmap: for_mkfs=false 2022-03-29T17:49:53.440 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,353 [shard 0] monc - reopen_session to mon.-1 2022-03-29T17:49:53.440 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,353 [shard 0] monc - connecting to mon.1 2022-03-29T17:49:53.441 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,353 [shard 0] ms - [client.?(temp_mon_client) - >> mon.? v2:172.21.15.138:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:3300/0, peer_name=mon.?, cc=216034701559232259 policy(lossy=false, server=false, standby=false, resetcheck=true) 2022-03-29T17:49:53.441 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,354 [shard 0] ms - [client.?(temp_mon_client) - >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:49:53.441 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,354 [shard 0] monc - mon.0 does not have an addr compatible with me 2022-03-29T17:49:53.441 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,355 [shard 0] ms - [client.?(temp_mon_client) -@57921 >> mon.? v2:172.21.15.138:3300/0] peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:57921/0 (socket says 172.21.15.138:57921) 2022-03-29T17:49:53.442 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,356 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35578@57921 >> mon.? v2:172.21.15.138:3300/0] learned myaddr=172.21.15.138:0/35578 (unbound) 2022-03-29T17:49:53.442 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,356 [shard 0] monc - get_auth_request(con=[client.?(temp_mon_client) 172.21.15.138:0/35578@57921 >> mon.? v2:172.21.15.138:3300/0], auth_method=0) 2022-03-29T17:49:53.442 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,356 [shard 0] monc - method 2 preferred_modes {1} 2022-03-29T17:49:53.442 INFO:tasks.ceph.osd.2.smithi138.stdout:WARNING: debug mode. Not for benchmarking or production 2022-03-29T17:49:53.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,367 [shard 0] osd - open_all_classes 2022-03-29T17:49:53.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,367 [shard 0] osd - open_all_classes found journal 2022-03-29T17:49:53.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,367 [shard 0] osd - _get_class adding new class name journal 0x612000025d00 2022-03-29T17:49:53.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,367 [shard 0] osd - _load_class journal from /usr/lib64/rados-classes/libcls_journal.so 2022-03-29T17:49:53.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,367 [shard 0] osd - register_class journal status 3 2022-03-29T17:49:53.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,367 [shard 0] osd - register_cxx_method journal.create flags 3 0x7f0d5875a7de 2022-03-29T17:49:53.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,367 [shard 0] osd - register_cxx_method journal.get_order flags 1 0x7f0d5875a1de 2022-03-29T17:49:53.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,367 [shard 0] osd - register_cxx_method journal.get_splay_width flags 1 0x7f0d5875a246 2022-03-29T17:49:53.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.get_pool_id flags 1 0x7f0d5875ac38 2022-03-29T17:49:53.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.get_minimum_set flags 1 0x7f0d5875ae60 2022-03-29T17:49:53.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.set_minimum_set flags 3 0x7f0d5875afb6 2022-03-29T17:49:53.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.get_active_set flags 1 0x7f0d5875aed2 2022-03-29T17:49:53.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.set_active_set flags 3 0x7f0d5875b155 2022-03-29T17:49:53.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.get_client flags 1 0x7f0d5875c4e4 2022-03-29T17:49:53.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.client_register flags 3 0x7f0d5875dc5e 2022-03-29T17:49:53.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.client_update_data flags 3 0x7f0d5875c7bf 2022-03-29T17:49:53.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.client_update_state flags 3 0x7f0d5875cc9f 2022-03-29T17:49:53.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.client_unregister flags 3 0x7f0d5875fe8a 2022-03-29T17:49:53.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.client_commit flags 3 0x7f0d58760832 2022-03-29T17:49:53.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.client_list flags 1 0x7f0d5875e30f 2022-03-29T17:49:53.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.get_next_tag_tid flags 1 0x7f0d5875af44 2022-03-29T17:49:53.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.get_tag flags 1 0x7f0d5875bf5d 2022-03-29T17:49:53.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.tag_create flags 3 0x7f0d58760129 2022-03-29T17:49:53.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.tag_list flags 1 0x7f0d5875e573 2022-03-29T17:49:53.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.guard_append flags 3 0x7f0d58759b5c 2022-03-29T17:49:53.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - register_cxx_method journal.append flags 3 0x7f0d5875b2f4 2022-03-29T17:49:53.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - _load_class journal success 2022-03-29T17:49:53.488 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - open_all_classes found rgw 2022-03-29T17:49:53.488 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - _get_class adding new class name rgw 0x612000025e80 2022-03-29T17:49:53.488 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,368 [shard 0] osd - _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so 2022-03-29T17:49:53.488 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,369 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw/cls_rgw.cc:4353: Loaded rgw class! 2022-03-29T17:49:53.488 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_class rgw status 3 2022-03-29T17:49:53.489 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_cxx_method rgw.bucket_init_index flags 3 0x7f0d5840f175 2022-03-29T17:49:53.489 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f0d5841eca1 2022-03-29T17:49:53.489 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_cxx_method rgw.bucket_list flags 1 0x7f0d5842b331 2022-03-29T17:49:53.489 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_cxx_method rgw.bucket_check_index flags 1 0x7f0d58429ea4 2022-03-29T17:49:53.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f0d5842a70f 2022-03-29T17:49:53.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_cxx_method rgw.bucket_update_stats flags 3 0x7f0d584205bd 2022-03-29T17:49:53.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_cxx_method rgw.bucket_prepare_op flags 3 0x7f0d5841a3dd 2022-03-29T17:49:53.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,369 [shard 0] osd - register_cxx_method rgw.bucket_complete_op flags 3 0x7f0d5842225b 2022-03-29T17:49:53.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bucket_link_olh flags 3 0x7f0d5842d49b 2022-03-29T17:49:53.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f0d58424434 2022-03-29T17:49:53.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f0d58425d50 2022-03-29T17:49:53.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f0d58423dcb 2022-03-29T17:49:53.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bucket_clear_olh flags 3 0x7f0d5842361e 2022-03-29T17:49:53.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.obj_remove flags 3 0x7f0d5842748a 2022-03-29T17:49:53.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.obj_store_pg_ver flags 2 0x7f0d58413019 2022-03-29T17:49:53.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f0d58412cda 2022-03-29T17:49:53.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.obj_check_mtime flags 1 0x7f0d58412a18 2022-03-29T17:49:53.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bi_get flags 1 0x7f0d584124fe 2022-03-29T17:49:53.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bi_put flags 3 0x7f0d58412304 2022-03-29T17:49:53.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bi_list flags 1 0x7f0d58428b1e 2022-03-29T17:49:53.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bi_log_list flags 1 0x7f0d58419f1a 2022-03-29T17:49:53.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bi_log_trim flags 3 0x7f0d58411d56 2022-03-29T17:49:53.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.dir_suggest_changes flags 3 0x7f0d5842093a 2022-03-29T17:49:53.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bi_log_resync flags 3 0x7f0d5841ffe3 2022-03-29T17:49:53.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.bi_log_stop flags 3 0x7f0d5841fa09 2022-03-29T17:49:53.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.user_usage_log_add flags 3 0x7f0d5841be48 2022-03-29T17:49:53.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.user_usage_log_read flags 1 0x7f0d5841b6db 2022-03-29T17:49:53.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.user_usage_log_trim flags 3 0x7f0d5841bbf9 2022-03-29T17:49:53.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.usage_log_clear flags 2 0x7f0d5840cdda 2022-03-29T17:49:53.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.gc_set_entry flags 3 0x7f0d5841538e 2022-03-29T17:49:53.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.gc_defer_entry flags 3 0x7f0d58415244 2022-03-29T17:49:53.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.gc_list flags 1 0x7f0d584139c9 2022-03-29T17:49:53.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.gc_remove flags 3 0x7f0d5842aeb9 2022-03-29T17:49:53.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.lc_get_entry flags 1 0x7f0d58414423 2022-03-29T17:49:53.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.lc_set_entry flags 3 0x7f0d58416271 2022-03-29T17:49:53.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.lc_rm_entry flags 3 0x7f0d58416122 2022-03-29T17:49:53.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.lc_get_next_entry flags 1 0x7f0d58411850 2022-03-29T17:49:53.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.lc_put_head flags 3 0x7f0d584115fb 2022-03-29T17:49:53.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.lc_get_head flags 1 0x7f0d584112c1 2022-03-29T17:49:53.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.lc_list_entries flags 1 0x7f0d584183ee 2022-03-29T17:49:53.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.reshard_add flags 3 0x7f0d58410cbc 2022-03-29T17:49:53.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.reshard_list flags 1 0x7f0d58415941 2022-03-29T17:49:53.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.reshard_get flags 1 0x7f0d584105af 2022-03-29T17:49:53.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.reshard_remove flags 3 0x7f0d58410188 2022-03-29T17:49:53.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.set_bucket_resharding flags 3 0x7f0d5841f74a 2022-03-29T17:49:53.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.clear_bucket_resharding flags 3 0x7f0d5841f4ee 2022-03-29T17:49:53.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.guard_bucket_resharding flags 1 0x7f0d5841f291 2022-03-29T17:49:53.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method rgw.get_bucket_resharding flags 1 0x7f0d5841eeea 2022-03-29T17:49:53.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _load_class rgw success 2022-03-29T17:49:53.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - open_all_classes found cephfs 2022-03-29T17:49:53.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _get_class adding new class name cephfs 0x612000026180 2022-03-29T17:49:53.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _load_class cephfs from /usr/lib64/rados-classes/libcls_cephfs.so 2022-03-29T17:49:53.500 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,370 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cephfs/cls_cephfs.cc:201: loading cephfs 2022-03-29T17:49:53.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_class cephfs status 3 2022-03-29T17:49:53.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method cephfs.accumulate_inode_metadata flags 3 0x7f0d5815a2f9 2022-03-29T17:49:53.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _load_class cephfs success 2022-03-29T17:49:53.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - open_all_classes found numops 2022-03-29T17:49:53.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _get_class adding new class name numops 0x612000026480 2022-03-29T17:49:53.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _load_class numops from /usr/lib64/rados-classes/libcls_numops.so 2022-03-29T17:49:53.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_class numops status 3 2022-03-29T17:49:53.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method numops.add flags 3 0x7f0d57f4c65f 2022-03-29T17:49:53.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - register_cxx_method numops.mul flags 3 0x7f0d57f4bbe6 2022-03-29T17:49:53.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _load_class numops success 2022-03-29T17:49:53.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - open_all_classes found lua 2022-03-29T17:49:53.503 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,370 [shard 0] osd - _get_class not permitted to load lua 2022-03-29T17:49:53.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - open_all_classes found test_remote_reads 2022-03-29T17:49:53.503 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,370 [shard 0] osd - _get_class not permitted to load test_remote_reads 2022-03-29T17:49:53.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - open_all_classes found rbd 2022-03-29T17:49:53.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _get_class adding new class name rbd 0x612000026780 2022-03-29T17:49:53.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,370 [shard 0] osd - _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so 2022-03-29T17:49:53.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_class rbd status 3 2022-03-29T17:49:53.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.create flags 3 0x7f0d57cecec0 2022-03-29T17:49:53.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_features flags 1 0x7f0d57cda5af 2022-03-29T17:49:53.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_features flags 3 0x7f0d57cda7d2 2022-03-29T17:49:53.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_size flags 1 0x7f0d57cffcb9 2022-03-29T17:49:53.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_size flags 3 0x7f0d57cdf75d 2022-03-29T17:49:53.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_snapcontext flags 1 0x7f0d57cdb27a 2022-03-29T17:49:53.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_object_prefix flags 1 0x7f0d57cd9aaf 2022-03-29T17:49:53.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_data_pool flags 1 0x7f0d57cd95be 2022-03-29T17:49:53.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_snapshot_name flags 1 0x7f0d57cf8f69 2022-03-29T17:49:53.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_snapshot_timestamp flags 1 0x7f0d57cf8bc7 2022-03-29T17:49:53.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snapshot_get flags 1 0x7f0d57cfddcd 2022-03-29T17:49:53.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snapshot_add flags 3 0x7f0d57d049a5 2022-03-29T17:49:53.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snapshot_remove flags 3 0x7f0d57cfd0e4 2022-03-29T17:49:53.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snapshot_rename flags 3 0x7f0d57cfa938 2022-03-29T17:49:53.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snapshot_trash_add flags 3 0x7f0d57cf982f 2022-03-29T17:49:53.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_all_features flags 1 0x7f0d57cd00bc 2022-03-29T17:49:53.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_parent flags 1 0x7f0d57cf7d9f 2022-03-29T17:49:53.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_parent flags 3 0x7f0d57cea007 2022-03-29T17:49:53.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.remove_parent flags 3 0x7f0d57cfca52 2022-03-29T17:49:53.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.parent_get flags 1 0x7f0d57cf891f 2022-03-29T17:49:53.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.parent_overlap_get flags 1 0x7f0d57d00270 2022-03-29T17:49:53.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.parent_attach flags 3 0x7f0d57ce0489 2022-03-29T17:49:53.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.parent_detach flags 3 0x7f0d57cfca72 2022-03-29T17:49:53.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_protection_status flags 3 0x7f0d57cff7f9 2022-03-29T17:49:53.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_protection_status flags 1 0x7f0d57cfe655 2022-03-29T17:49:53.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f0d57cdd140 2022-03-29T17:49:53.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f0d57cdc5c9 2022-03-29T17:49:53.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_create_timestamp flags 1 0x7f0d57cd9e65 2022-03-29T17:49:53.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_access_timestamp flags 1 0x7f0d57cda0d3 2022-03-29T17:49:53.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_modify_timestamp flags 1 0x7f0d57cda341 2022-03-29T17:49:53.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_flags flags 1 0x7f0d57d00842 2022-03-29T17:49:53.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_flags flags 3 0x7f0d57cfca92 2022-03-29T17:49:53.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.op_features_get flags 1 0x7f0d57cd944f 2022-03-29T17:49:53.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.op_features_set flags 3 0x7f0d57cdda28 2022-03-29T17:49:53.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.metadata_list flags 1 0x7f0d57ceed62 2022-03-29T17:49:53.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.metadata_set flags 3 0x7f0d57cf75a7 2022-03-29T17:49:53.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.metadata_remove flags 3 0x7f0d57cd6e87 2022-03-29T17:49:53.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.metadata_get flags 1 0x7f0d57cd7065 2022-03-29T17:49:53.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snapshot_get_limit flags 1 0x7f0d57cd9731 2022-03-29T17:49:53.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snapshot_set_limit flags 3 0x7f0d57cdc079 2022-03-29T17:49:53.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.child_attach flags 3 0x7f0d57cfd73a 2022-03-29T17:49:53.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.child_detach flags 3 0x7f0d57cff0fc 2022-03-29T17:49:53.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.children_list flags 1 0x7f0d57cf9329 2022-03-29T17:49:53.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.migration_set flags 3 0x7f0d57cdeb72 2022-03-29T17:49:53.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.migration_set_state flags 3 0x7f0d57cdedc9 2022-03-29T17:49:53.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.migration_get flags 1 0x7f0d57cdb075 2022-03-29T17:49:53.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.migration_remove flags 3 0x7f0d57cdd125 2022-03-29T17:49:53.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_modify_timestamp flags 3 0x7f0d57cd99aa 2022-03-29T17:49:53.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_access_timestamp flags 3 0x7f0d57cd98a5 2022-03-29T17:49:53.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.add_child flags 3 0x7f0d57d01af5 2022-03-29T17:49:53.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.remove_child flags 3 0x7f0d57d02052 2022-03-29T17:49:53.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_children flags 1 0x7f0d57d02633 2022-03-29T17:49:53.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.get_id flags 1 0x7f0d57cd248d 2022-03-29T17:49:53.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.set_id flags 3 0x7f0d57cd26f0 2022-03-29T17:49:53.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.dir_get_id flags 1 0x7f0d57cd5c9d 2022-03-29T17:49:53.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.dir_get_name flags 1 0x7f0d57cd659c 2022-03-29T17:49:53.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.dir_list flags 1 0x7f0d57cec5f8 2022-03-29T17:49:53.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.dir_add_image flags 3 0x7f0d57cf0a41 2022-03-29T17:49:53.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.dir_remove_image flags 3 0x7f0d57cd641b 2022-03-29T17:49:53.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.dir_rename_image flags 3 0x7f0d57cf080d 2022-03-29T17:49:53.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.dir_state_assert flags 1 0x7f0d57cd933d 2022-03-29T17:49:53.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.dir_state_set flags 3 0x7f0d57cdbdf0 2022-03-29T17:49:53.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.object_map_load flags 1 0x7f0d57cf0dee 2022-03-29T17:49:53.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.object_map_save flags 3 0x7f0d57cf12fc 2022-03-29T17:49:53.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.object_map_resize flags 3 0x7f0d57cf2a9b 2022-03-29T17:49:53.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.object_map_update flags 3 0x7f0d57cf1bce 2022-03-29T17:49:53.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.object_map_snap_add flags 3 0x7f0d57cf0ec6 2022-03-29T17:49:53.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.object_map_snap_remove flags 3 0x7f0d57cf14bf 2022-03-29T17:49:53.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snap_list flags 1 0x7f0d57cd8f91 2022-03-29T17:49:53.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snap_add flags 3 0x7f0d57cdb68e 2022-03-29T17:49:53.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snap_remove flags 3 0x7f0d57cd2b41 2022-03-29T17:49:53.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.snap_rename flags 3 0x7f0d57cd32a2 2022-03-29T17:49:53.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_uuid_get flags 1 0x7f0d57cd8cef 2022-03-29T17:49:53.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_uuid_set flags 3 0x7f0d57ce5253 2022-03-29T17:49:53.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_mode_get flags 1 0x7f0d57cd3c4b 2022-03-29T17:49:53.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_mode_set flags 3 0x7f0d57ce8ba6 2022-03-29T17:49:53.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_peer_ping flags 3 0x7f0d57cf73fb 2022-03-29T17:49:53.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_peer_list flags 1 0x7f0d57ce9035 2022-03-29T17:49:53.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_peer_add flags 3 0x7f0d57cea2de 2022-03-29T17:49:53.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_peer_remove flags 3 0x7f0d57ce113d 2022-03-29T17:49:53.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_client flags 3 0x7f0d57ce0b04 2022-03-29T17:49:53.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_cluster flags 3 0x7f0d57ce918f 2022-03-29T17:49:53.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_direction flags 3 0x7f0d57ce0ddb 2022-03-29T17:49:53.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_list flags 1 0x7f0d57cebcae 2022-03-29T17:49:53.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_get_image_id flags 1 0x7f0d57ce1e8c 2022-03-29T17:49:53.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_get flags 1 0x7f0d57ce149a 2022-03-29T17:49:53.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_set flags 3 0x7f0d57ce1d0f 2022-03-29T17:49:53.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_remove flags 3 0x7f0d57ce7f25 2022-03-29T17:49:53.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_status_set flags 11 0x7f0d57ce77da 2022-03-29T17:49:53.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove flags 3 0x7f0d57ce802f 2022-03-29T17:49:53.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get flags 1 0x7f0d57ce9e5f 2022-03-29T17:49:53.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,371 [shard 0] osd - register_cxx_method rbd.mirror_image_status_list flags 1 0x7f0d57cf3d7f 2022-03-29T17:49:53.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get_summary flags 1 0x7f0d57cf4570 2022-03-29T17:49:53.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove_down flags 3 0x7f0d57ce6f42 2022-03-29T17:49:53.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_get flags 1 0x7f0d57ce72cd 2022-03-29T17:49:53.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_list flags 1 0x7f0d57cf504d 2022-03-29T17:49:53.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_instances_list flags 1 0x7f0d57ce5851 2022-03-29T17:49:53.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_instances_add flags 11 0x7f0d57ce24de 2022-03-29T17:49:53.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_instances_remove flags 3 0x7f0d57ce26ef 2022-03-29T17:49:53.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_map_list flags 1 0x7f0d57ce505e 2022-03-29T17:49:53.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_map_update flags 2 0x7f0d57ce2841 2022-03-29T17:49:53.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_map_remove flags 2 0x7f0d57ce2bbd 2022-03-29T17:49:53.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_unlink_peer flags 3 0x7f0d57d03b3c 2022-03-29T17:49:53.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_set_copy_progress flags 3 0x7f0d57d02e49 2022-03-29T17:49:53.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_dir_list flags 1 0x7f0d57ceb196 2022-03-29T17:49:53.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_dir_add flags 3 0x7f0d57cefca7 2022-03-29T17:49:53.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_dir_remove flags 3 0x7f0d57cd6cc2 2022-03-29T17:49:53.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_dir_rename flags 3 0x7f0d57cefec2 2022-03-29T17:49:53.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_image_remove flags 3 0x7f0d57cd032a 2022-03-29T17:49:53.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_image_list flags 1 0x7f0d57ce5c9a 2022-03-29T17:49:53.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_image_set flags 3 0x7f0d57cd3cb3 2022-03-29T17:49:53.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.image_group_add flags 3 0x7f0d57cddb03 2022-03-29T17:49:53.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.image_group_remove flags 3 0x7f0d57cddf46 2022-03-29T17:49:53.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.image_group_get flags 1 0x7f0d57cd9c65 2022-03-29T17:49:53.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_snap_set flags 3 0x7f0d57cf5ae6 2022-03-29T17:49:53.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_snap_remove flags 3 0x7f0d57cd7b76 2022-03-29T17:49:53.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_snap_get_by_id flags 1 0x7f0d57cd7d09 2022-03-29T17:49:53.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.group_snap_list flags 1 0x7f0d57ceac44 2022-03-29T17:49:53.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.trash_add flags 3 0x7f0d57cee79b 2022-03-29T17:49:53.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.trash_remove flags 3 0x7f0d57ce2de3 2022-03-29T17:49:53.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.trash_list flags 1 0x7f0d57cf5ece 2022-03-29T17:49:53.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.trash_get flags 1 0x7f0d57ce30e8 2022-03-29T17:49:53.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.trash_state_set flags 3 0x7f0d57ce336e 2022-03-29T17:49:53.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.namespace_add flags 3 0x7f0d57ce37a6 2022-03-29T17:49:53.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.namespace_remove flags 3 0x7f0d57ce39d1 2022-03-29T17:49:53.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.namespace_list flags 1 0x7f0d57ce3bba 2022-03-29T17:49:53.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.copyup flags 3 0x7f0d57cd016b 2022-03-29T17:49:53.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.sparse_copyup flags 3 0x7f0d57d00e66 2022-03-29T17:49:53.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.assert_snapc_seq flags 3 0x7f0d57cd021b 2022-03-29T17:49:53.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method rbd.sparsify flags 3 0x7f0d57cd4074 2022-03-29T17:49:53.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - _load_class rbd success 2022-03-29T17:49:53.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - open_all_classes found queue 2022-03-29T17:49:53.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - _get_class adding new class name queue 0x612000026900 2022-03-29T17:49:53.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - _load_class queue from /usr/lib64/rados-classes/libcls_queue.so 2022-03-29T17:49:53.534 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,372 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/queue/cls_queue.cc:125: Loaded queue class! 2022-03-29T17:49:53.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_class queue status 3 2022-03-29T17:49:53.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method queue.queue_init flags 3 0x7f0d57a94256 2022-03-29T17:49:53.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method queue.queue_get_capacity flags 1 0x7f0d57a939e9 2022-03-29T17:49:53.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method queue.queue_enqueue flags 3 0x7f0d57a9452e 2022-03-29T17:49:53.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method queue.queue_list_entries flags 1 0x7f0d57a93daf 2022-03-29T17:49:53.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method queue.queue_remove_entries flags 3 0x7f0d57a93b81 2022-03-29T17:49:53.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - _load_class queue success 2022-03-29T17:49:53.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - open_all_classes found user 2022-03-29T17:49:53.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - _get_class adding new class name user 0x612000026c00 2022-03-29T17:49:53.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - _load_class user from /usr/lib64/rados-classes/libcls_user.so 2022-03-29T17:49:53.537 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,372 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/user/cls_user.cc:506: Loaded user class! 2022-03-29T17:49:53.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_class user status 3 2022-03-29T17:49:53.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method user.set_buckets_info flags 3 0x7f0d5787ac4b 2022-03-29T17:49:53.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method user.complete_stats_sync flags 3 0x7f0d5787a9a3 2022-03-29T17:49:53.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method user.remove_bucket flags 3 0x7f0d5787a1be 2022-03-29T17:49:53.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method user.list_buckets flags 1 0x7f0d5787b733 2022-03-29T17:49:53.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,372 [shard 0] osd - register_cxx_method user.get_header flags 1 0x7f0d57879fc7 2022-03-29T17:49:53.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method user.reset_user_stats flags 3 0x7f0d5787954c 2022-03-29T17:49:53.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method user.reset_user_stats2 flags 3 0x7f0d5787c19b 2022-03-29T17:49:53.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _load_class user success 2022-03-29T17:49:53.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - open_all_classes found cas 2022-03-29T17:49:53.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _get_class adding new class name cas 0x612000027080 2022-03-29T17:49:53.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _load_class cas from /usr/lib64/rados-classes/libcls_cas.so 2022-03-29T17:49:53.540 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,373 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cas/cls_cas.cc:211: Loaded cas class! 2022-03-29T17:49:53.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_class cas status 3 2022-03-29T17:49:53.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method cas.chunk_create_or_get_ref flags 3 0x7f0d57663b38 2022-03-29T17:49:53.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method cas.chunk_get_ref flags 3 0x7f0d57663848 2022-03-29T17:49:53.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method cas.chunk_put_ref flags 3 0x7f0d576634c4 2022-03-29T17:49:53.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method cas.references_chunk flags 1 0x7f0d576631fc 2022-03-29T17:49:53.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _load_class cas success 2022-03-29T17:49:53.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - open_all_classes found fifo 2022-03-29T17:49:53.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _get_class adding new class name fifo 0x612000027380 2022-03-29T17:49:53.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _load_class fifo from /usr/lib64/rados-classes/libcls_fifo.so 2022-03-29T17:49:53.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_class fifo status 3 2022-03-29T17:49:53.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method fifo.create_meta flags 3 0x7f0d57424dee 2022-03-29T17:49:53.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method fifo.get_meta flags 1 0x7f0d574260b2 2022-03-29T17:49:53.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method fifo.update_meta flags 3 0x7f0d5742657b 2022-03-29T17:49:53.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method fifo.init_part flags 3 0x7f0d57423cf2 2022-03-29T17:49:53.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method fifo.push_part flags 3 0x7f0d574243e5 2022-03-29T17:49:53.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method fifo.trim_part flags 3 0x7f0d57423782 2022-03-29T17:49:53.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method fifo.part_list flags 1 0x7f0d57422ed9 2022-03-29T17:49:53.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method fifo.get_part_info flags 1 0x7f0d5742277e 2022-03-29T17:49:53.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _load_class fifo success 2022-03-29T17:49:53.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - open_all_classes found cmpomap 2022-03-29T17:49:53.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _get_class adding new class name cmpomap 0x612000027680 2022-03-29T17:49:53.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _load_class cmpomap from /usr/lib64/rados-classes/libcls_cmpomap.so 2022-03-29T17:49:53.545 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,373 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cmpomap/server.cc:287: Loaded cmpomap class! 2022-03-29T17:49:53.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_class cmpomap status 3 2022-03-29T17:49:53.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method cmpomap.cmp_vals flags 1 0x7f0d572072e7 2022-03-29T17:49:53.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method cmpomap.cmp_set_vals flags 3 0x7f0d5720670e 2022-03-29T17:49:53.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - register_cxx_method cmpomap.cmp_rm_keys flags 3 0x7f0d57206136 2022-03-29T17:49:53.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _load_class cmpomap success 2022-03-29T17:49:53.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - open_all_classes found hello 2022-03-29T17:49:53.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _get_class adding new class name hello 0x612000027b00 2022-03-29T17:49:53.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,373 [shard 0] osd - _load_class hello from /usr/lib64/rados-classes/libcls_hello.so 2022-03-29T17:49:53.547 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,374 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/hello/cls_hello.cc:316: loading cls_hello 2022-03-29T17:49:53.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_class hello status 3 2022-03-29T17:49:53.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.say_hello flags 1 0x7f0d56ff6bf6 2022-03-29T17:49:53.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.record_hello flags 10 0x7f0d56ff6ea9 2022-03-29T17:49:53.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.write_return_data flags 2 0x7f0d56ff6d7f 2022-03-29T17:49:53.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.writes_dont_return_data flags 2 0x7f0d56ff6d7f 2022-03-29T17:49:53.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.write_too_much_return_data flags 2 0x7f0d56ff6c6d 2022-03-29T17:49:53.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.replay flags 1 0x7f0d56ff6bce 2022-03-29T17:49:53.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.turn_it_to_11 flags 11 0x7f0d56ff7808 2022-03-29T17:49:53.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.bad_reader flags 2 0x7f0d56ff6baf 2022-03-29T17:49:53.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method hello.bad_writer flags 1 0x7f0d56ff6b9d 2022-03-29T17:49:53.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _load_class hello success 2022-03-29T17:49:53.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - open_all_classes found log 2022-03-29T17:49:53.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _get_class adding new class name log 0x612000027e00 2022-03-29T17:49:53.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _load_class log from /usr/lib64/rados-classes/libcls_log.so 2022-03-29T17:49:53.550 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,374 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/log/cls_log.cc:305: Loaded log class! 2022-03-29T17:49:53.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_class log status 3 2022-03-29T17:49:53.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method log.add flags 3 0x7f0d56de5f1f 2022-03-29T17:49:53.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method log.list flags 1 0x7f0d56de6a5d 2022-03-29T17:49:53.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method log.trim flags 3 0x7f0d56de62cf 2022-03-29T17:49:53.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method log.info flags 1 0x7f0d56de5af7 2022-03-29T17:49:53.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _load_class log success 2022-03-29T17:49:53.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - open_all_classes found otp 2022-03-29T17:49:53.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _get_class adding new class name otp 0x612000028280 2022-03-29T17:49:53.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _load_class otp from /usr/lib64/rados-classes/libcls_otp.so 2022-03-29T17:49:53.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_class otp status 3 2022-03-29T17:49:53.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method otp.otp_set flags 3 0x7f0d56bd237f 2022-03-29T17:49:53.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method otp.otp_get flags 1 0x7f0d56bd198f 2022-03-29T17:49:53.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method otp.otp_check flags 3 0x7f0d56bd0fc3 2022-03-29T17:49:53.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method otp.otp_get_result flags 1 0x7f0d56bd0902 2022-03-29T17:49:53.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method otp.otp_remove flags 3 0x7f0d56bd1700 2022-03-29T17:49:53.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - register_cxx_method otp.get_current_time flags 1 0x7f0d56bd05f0 2022-03-29T17:49:53.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _load_class otp success 2022-03-29T17:49:53.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - open_all_classes found sdk 2022-03-29T17:49:53.554 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,374 [shard 0] osd - _get_class not permitted to load sdk 2022-03-29T17:49:53.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - open_all_classes found timeindex 2022-03-29T17:49:53.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _get_class adding new class name timeindex 0x612000028400 2022-03-29T17:49:53.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,374 [shard 0] osd - _load_class timeindex from /usr/lib64/rados-classes/libcls_timeindex.so 2022-03-29T17:49:53.555 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,375 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/timeindex/cls_timeindex.cc:247: Loaded timeindex class! 2022-03-29T17:49:53.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_class timeindex status 3 2022-03-29T17:49:53.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method timeindex.add flags 3 0x7f0d567a3894 2022-03-29T17:49:53.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method timeindex.list flags 1 0x7f0d567a3c23 2022-03-29T17:49:53.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method timeindex.trim flags 3 0x7f0d567a343d 2022-03-29T17:49:53.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _load_class timeindex success 2022-03-29T17:49:53.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - open_all_classes found lock 2022-03-29T17:49:53.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _get_class adding new class name lock 0x612000028880 2022-03-29T17:49:53.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _load_class lock from /usr/lib64/rados-classes/libcls_lock.so 2022-03-29T17:49:53.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_class lock status 3 2022-03-29T17:49:53.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method lock.lock flags 11 0x7f0d565904af 2022-03-29T17:49:53.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method lock.unlock flags 11 0x7f0d5658de27 2022-03-29T17:49:53.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method lock.break_lock flags 3 0x7f0d5658dccd 2022-03-29T17:49:53.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method lock.get_info flags 1 0x7f0d5658ea33 2022-03-29T17:49:53.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method lock.list_locks flags 1 0x7f0d5658c26b 2022-03-29T17:49:53.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method lock.assert_locked flags 9 0x7f0d5658d3bd 2022-03-29T17:49:53.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method lock.set_cookie flags 11 0x7f0d5658dff4 2022-03-29T17:49:53.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _load_class lock success 2022-03-29T17:49:53.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - open_all_classes found 2pc_queue 2022-03-29T17:49:53.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _get_class adding new class name 2pc_queue 0x612000028a00 2022-03-29T17:49:53.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _load_class 2pc_queue from /usr/lib64/rados-classes/libcls_2pc_queue.so 2022-03-29T17:49:53.560 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,375 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/2pc_queue/cls_2pc_queue.cc:575: Loaded 2pc queue class! 2022-03-29T17:49:53.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_class 2pc_queue status 3 2022-03-29T17:49:53.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_init flags 3 0x7f0d5634f6a3 2022-03-29T17:49:53.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_get_capacity flags 1 0x7f0d5634ec19 2022-03-29T17:49:53.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_reserve flags 3 0x7f0d56352504 2022-03-29T17:49:53.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_commit flags 3 0x7f0d5634fb0d 2022-03-29T17:49:53.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_abort flags 3 0x7f0d5635128f 2022-03-29T17:49:53.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_reservations flags 1 0x7f0d56351c8c 2022-03-29T17:49:53.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_entries flags 1 0x7f0d5634f076 2022-03-29T17:49:53.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_remove_entries flags 3 0x7f0d5634ee39 2022-03-29T17:49:53.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_expire_reservations flags 3 0x7f0d563507d0 2022-03-29T17:49:53.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _load_class 2pc_queue success 2022-03-29T17:49:53.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - open_all_classes found refcount 2022-03-29T17:49:53.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _get_class adding new class name refcount 0x612000028d00 2022-03-29T17:49:53.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,375 [shard 0] osd - _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so 2022-03-29T17:49:53.563 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,377 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/refcount/cls_refcount.cc:199: Loaded refcount class! 2022-03-29T17:49:53.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_class refcount status 3 2022-03-29T17:49:53.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method refcount.get flags 3 0x7f0d56087c89 2022-03-29T17:49:53.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method refcount.put flags 3 0x7f0d5608794c 2022-03-29T17:49:53.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method refcount.set flags 3 0x7f0d560870c1 2022-03-29T17:49:53.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method refcount.read flags 1 0x7f0d56087602 2022-03-29T17:49:53.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - _load_class refcount success 2022-03-29T17:49:53.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - open_all_classes found version 2022-03-29T17:49:53.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - _get_class adding new class name version 0x612000029000 2022-03-29T17:49:53.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - _load_class version from /usr/lib64/rados-classes/libcls_version.so 2022-03-29T17:49:53.566 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,377 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/version/cls_version.cc:218: Loaded version class! 2022-03-29T17:49:53.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_class version status 3 2022-03-29T17:49:53.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method version.set flags 3 0x7f0d55dd63d7 2022-03-29T17:49:53.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method version.inc flags 3 0x7f0d55dd66e1 2022-03-29T17:49:53.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method version.inc_conds flags 3 0x7f0d55dd66e1 2022-03-29T17:49:53.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method version.read flags 1 0x7f0d55dd621c 2022-03-29T17:49:53.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method version.check_conds flags 1 0x7f0d55dd64f9 2022-03-29T17:49:53.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - _load_class version success 2022-03-29T17:49:53.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - open_all_classes found rgw_gc 2022-03-29T17:49:53.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - _get_class adding new class name rgw_gc 0x612000029480 2022-03-29T17:49:53.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - _load_class rgw_gc from /usr/lib64/rados-classes/libcls_rgw_gc.so 2022-03-29T17:49:53.568 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,377 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw_gc/cls_rgw_gc.cc:539: Loaded rgw gc class! 2022-03-29T17:49:53.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_class rgw_gc status 3 2022-03-29T17:49:53.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_init flags 3 0x7f0d55b99a31 2022-03-29T17:49:53.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_enqueue flags 3 0x7f0d55b99130 2022-03-29T17:49:53.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_list_entries flags 1 0x7f0d55b9b2eb 2022-03-29T17:49:53.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_remove_entries flags 3 0x7f0d55b99d97 2022-03-29T17:49:53.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_update_entry flags 3 0x7f0d55b9ce0e 2022-03-29T17:49:53.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,377 [shard 0] osd - _load_class rgw_gc success 2022-03-29T17:49:53.570 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,377 [shard 0] osd - OSD: nonce is 35579 2022-03-29T17:49:53.570 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding auth protocol: cephx 2022-03-29T17:49:53.570 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding auth protocol: cephx 2022-03-29T17:49:53.570 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding auth protocol: cephx 2022-03-29T17:49:53.571 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding con mode: crc 2022-03-29T17:49:53.571 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding con mode: crc 2022-03-29T17:49:53.571 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding con mode: crc 2022-03-29T17:49:53.571 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding con mode: crc 2022-03-29T17:49:53.572 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding con mode: crc 2022-03-29T17:49:53.572 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - AuthRegistry(0x7f0d55b8bb60) adding con mode: crc 2022-03-29T17:49:53.572 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,378 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring 2022-03-29T17:49:53.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,380 [shard 0] monc - build_monmap: for_mkfs=false 2022-03-29T17:49:53.572 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,380 [shard 0] monc - reopen_session to mon.-1 2022-03-29T17:49:53.573 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,380 [shard 0] monc - connecting to mon.1 2022-03-29T17:49:53.573 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,380 [shard 0] ms - [client.?(temp_mon_client) - >> mon.? v2:172.21.15.138:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:3300/0, peer_name=mon.?, cc=1072012829979169547 policy(lossy=false, server=false, standby=false, resetcheck=true) 2022-03-29T17:49:53.573 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,380 [shard 0] ms - [client.?(temp_mon_client) - >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:49:53.573 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,380 [shard 0] monc - mon.0 does not have an addr compatible with me 2022-03-29T17:49:53.573 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,381 [shard 0] ms - [client.?(temp_mon_client) -@57488 >> mon.? v2:172.21.15.138:3300/0] peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:57488/0 (socket says 172.21.15.138:57488) 2022-03-29T17:49:53.574 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,381 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35579@57488 >> mon.? v2:172.21.15.138:3300/0] learned myaddr=172.21.15.138:0/35579 (unbound) 2022-03-29T17:49:53.574 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,381 [shard 0] monc - get_auth_request(con=[client.?(temp_mon_client) 172.21.15.138:0/35579@57488 >> mon.? v2:172.21.15.138:3300/0], auth_method=0) 2022-03-29T17:49:53.574 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,381 [shard 0] monc - method 2 preferred_modes {1} 2022-03-29T17:49:53.575 INFO:tasks.ceph.osd.0.smithi138.stdout:WARNING: debug mode. Not for benchmarking or production 2022-03-29T17:49:53.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,391 [shard 0] osd - open_all_classes 2022-03-29T17:49:53.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,391 [shard 0] osd - open_all_classes found journal 2022-03-29T17:49:53.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,391 [shard 0] osd - _get_class adding new class name journal 0x612000025d00 2022-03-29T17:49:53.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,391 [shard 0] osd - _load_class journal from /usr/lib64/rados-classes/libcls_journal.so 2022-03-29T17:49:53.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_class journal status 3 2022-03-29T17:49:53.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.create flags 3 0x7fb5ffeb17de 2022-03-29T17:49:53.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.get_order flags 1 0x7fb5ffeb11de 2022-03-29T17:49:53.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.get_splay_width flags 1 0x7fb5ffeb1246 2022-03-29T17:49:53.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.get_pool_id flags 1 0x7fb5ffeb1c38 2022-03-29T17:49:53.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.get_minimum_set flags 1 0x7fb5ffeb1e60 2022-03-29T17:49:53.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.set_minimum_set flags 3 0x7fb5ffeb1fb6 2022-03-29T17:49:53.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.get_active_set flags 1 0x7fb5ffeb1ed2 2022-03-29T17:49:53.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.set_active_set flags 3 0x7fb5ffeb2155 2022-03-29T17:49:53.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.get_client flags 1 0x7fb5ffeb34e4 2022-03-29T17:49:53.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.client_register flags 3 0x7fb5ffeb4c5e 2022-03-29T17:49:53.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.client_update_data flags 3 0x7fb5ffeb37bf 2022-03-29T17:49:53.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.client_update_state flags 3 0x7fb5ffeb3c9f 2022-03-29T17:49:53.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.client_unregister flags 3 0x7fb5ffeb6e8a 2022-03-29T17:49:53.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.client_commit flags 3 0x7fb5ffeb7832 2022-03-29T17:49:53.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.client_list flags 1 0x7fb5ffeb530f 2022-03-29T17:49:53.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.get_next_tag_tid flags 1 0x7fb5ffeb1f44 2022-03-29T17:49:53.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.get_tag flags 1 0x7fb5ffeb2f5d 2022-03-29T17:49:53.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.tag_create flags 3 0x7fb5ffeb7129 2022-03-29T17:49:53.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.tag_list flags 1 0x7fb5ffeb5573 2022-03-29T17:49:53.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.guard_append flags 3 0x7fb5ffeb0b5c 2022-03-29T17:49:53.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - register_cxx_method journal.append flags 3 0x7fb5ffeb22f4 2022-03-29T17:49:53.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - _load_class journal success 2022-03-29T17:49:53.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - open_all_classes found rgw 2022-03-29T17:49:53.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - _get_class adding new class name rgw 0x612000025e80 2022-03-29T17:49:53.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,392 [shard 0] osd - _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so 2022-03-29T17:49:53.599 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,394 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw/cls_rgw.cc:4353: Loaded rgw class! 2022-03-29T17:49:53.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_class rgw status 3 2022-03-29T17:49:53.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_init_index flags 3 0x7fb5ffb66175 2022-03-29T17:49:53.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7fb5ffb75ca1 2022-03-29T17:49:53.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_list flags 1 0x7fb5ffb82331 2022-03-29T17:49:53.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_check_index flags 1 0x7fb5ffb80ea4 2022-03-29T17:49:53.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_rebuild_index flags 3 0x7fb5ffb8170f 2022-03-29T17:49:53.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_update_stats flags 3 0x7fb5ffb775bd 2022-03-29T17:49:53.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_prepare_op flags 3 0x7fb5ffb713dd 2022-03-29T17:49:53.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_complete_op flags 3 0x7fb5ffb7925b 2022-03-29T17:49:53.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_link_olh flags 3 0x7fb5ffb8449b 2022-03-29T17:49:53.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_unlink_instance flags 3 0x7fb5ffb7b434 2022-03-29T17:49:53.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_read_olh_log flags 1 0x7fb5ffb7cd50 2022-03-29T17:49:53.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7fb5ffb7adcb 2022-03-29T17:49:53.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bucket_clear_olh flags 3 0x7fb5ffb7a61e 2022-03-29T17:49:53.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.obj_remove flags 3 0x7fb5ffb7e48a 2022-03-29T17:49:53.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.obj_store_pg_ver flags 2 0x7fb5ffb6a019 2022-03-29T17:49:53.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7fb5ffb69cda 2022-03-29T17:49:53.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.obj_check_mtime flags 1 0x7fb5ffb69a18 2022-03-29T17:49:53.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bi_get flags 1 0x7fb5ffb694fe 2022-03-29T17:49:53.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bi_put flags 3 0x7fb5ffb69304 2022-03-29T17:49:53.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bi_list flags 1 0x7fb5ffb7fb1e 2022-03-29T17:49:53.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bi_log_list flags 1 0x7fb5ffb70f1a 2022-03-29T17:49:53.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bi_log_trim flags 3 0x7fb5ffb68d56 2022-03-29T17:49:53.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.dir_suggest_changes flags 3 0x7fb5ffb7793a 2022-03-29T17:49:53.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bi_log_resync flags 3 0x7fb5ffb76fe3 2022-03-29T17:49:53.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.bi_log_stop flags 3 0x7fb5ffb76a09 2022-03-29T17:49:53.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.user_usage_log_add flags 3 0x7fb5ffb72e48 2022-03-29T17:49:53.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.user_usage_log_read flags 1 0x7fb5ffb726db 2022-03-29T17:49:53.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.user_usage_log_trim flags 3 0x7fb5ffb72bf9 2022-03-29T17:49:53.606 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.usage_log_clear flags 2 0x7fb5ffb63dda 2022-03-29T17:49:53.606 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.gc_set_entry flags 3 0x7fb5ffb6c38e 2022-03-29T17:49:53.606 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.gc_defer_entry flags 3 0x7fb5ffb6c244 2022-03-29T17:49:53.606 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.gc_list flags 1 0x7fb5ffb6a9c9 2022-03-29T17:49:53.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.gc_remove flags 3 0x7fb5ffb81eb9 2022-03-29T17:49:53.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.lc_get_entry flags 1 0x7fb5ffb6b423 2022-03-29T17:49:53.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.lc_set_entry flags 3 0x7fb5ffb6d271 2022-03-29T17:49:53.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.lc_rm_entry flags 3 0x7fb5ffb6d122 2022-03-29T17:49:53.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.lc_get_next_entry flags 1 0x7fb5ffb68850 2022-03-29T17:49:53.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.lc_put_head flags 3 0x7fb5ffb685fb 2022-03-29T17:49:53.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.lc_get_head flags 1 0x7fb5ffb682c1 2022-03-29T17:49:53.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.lc_list_entries flags 1 0x7fb5ffb6f3ee 2022-03-29T17:49:53.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.reshard_add flags 3 0x7fb5ffb67cbc 2022-03-29T17:49:53.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.reshard_list flags 1 0x7fb5ffb6c941 2022-03-29T17:49:53.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.reshard_get flags 1 0x7fb5ffb675af 2022-03-29T17:49:53.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.reshard_remove flags 3 0x7fb5ffb67188 2022-03-29T17:49:53.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.set_bucket_resharding flags 3 0x7fb5ffb7674a 2022-03-29T17:49:53.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.clear_bucket_resharding flags 3 0x7fb5ffb764ee 2022-03-29T17:49:53.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.guard_bucket_resharding flags 1 0x7fb5ffb76291 2022-03-29T17:49:53.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - register_cxx_method rgw.get_bucket_resharding flags 1 0x7fb5ffb75eea 2022-03-29T17:49:53.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - _load_class rgw success 2022-03-29T17:49:53.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - open_all_classes found cephfs 2022-03-29T17:49:53.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - _get_class adding new class name cephfs 0x612000026180 2022-03-29T17:49:53.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,394 [shard 0] osd - _load_class cephfs from /usr/lib64/rados-classes/libcls_cephfs.so 2022-03-29T17:49:53.611 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,394 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cephfs/cls_cephfs.cc:201: loading cephfs 2022-03-29T17:49:53.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_class cephfs status 3 2022-03-29T17:49:53.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method cephfs.accumulate_inode_metadata flags 3 0x7fb5ff8b12f9 2022-03-29T17:49:53.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - _load_class cephfs success 2022-03-29T17:49:53.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - open_all_classes found numops 2022-03-29T17:49:53.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - _get_class adding new class name numops 0x612000026480 2022-03-29T17:49:53.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - _load_class numops from /usr/lib64/rados-classes/libcls_numops.so 2022-03-29T17:49:53.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_class numops status 3 2022-03-29T17:49:53.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method numops.add flags 3 0x7fb5ff6a365f 2022-03-29T17:49:53.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method numops.mul flags 3 0x7fb5ff6a2be6 2022-03-29T17:49:53.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - _load_class numops success 2022-03-29T17:49:53.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - open_all_classes found lua 2022-03-29T17:49:53.614 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,395 [shard 0] osd - _get_class not permitted to load lua 2022-03-29T17:49:53.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - open_all_classes found test_remote_reads 2022-03-29T17:49:53.614 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,395 [shard 0] osd - _get_class not permitted to load test_remote_reads 2022-03-29T17:49:53.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - open_all_classes found rbd 2022-03-29T17:49:53.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - _get_class adding new class name rbd 0x612000026780 2022-03-29T17:49:53.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so 2022-03-29T17:49:53.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_class rbd status 3 2022-03-29T17:49:53.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.create flags 3 0x7fb5ff443ec0 2022-03-29T17:49:53.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_features flags 1 0x7fb5ff4315af 2022-03-29T17:49:53.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.set_features flags 3 0x7fb5ff4317d2 2022-03-29T17:49:53.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_size flags 1 0x7fb5ff456cb9 2022-03-29T17:49:53.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.set_size flags 3 0x7fb5ff43675d 2022-03-29T17:49:53.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_snapcontext flags 1 0x7fb5ff43227a 2022-03-29T17:49:53.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_object_prefix flags 1 0x7fb5ff430aaf 2022-03-29T17:49:53.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_data_pool flags 1 0x7fb5ff4305be 2022-03-29T17:49:53.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_snapshot_name flags 1 0x7fb5ff44ff69 2022-03-29T17:49:53.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_snapshot_timestamp flags 1 0x7fb5ff44fbc7 2022-03-29T17:49:53.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.snapshot_get flags 1 0x7fb5ff454dcd 2022-03-29T17:49:53.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.snapshot_add flags 3 0x7fb5ff45b9a5 2022-03-29T17:49:53.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.snapshot_remove flags 3 0x7fb5ff4540e4 2022-03-29T17:49:53.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.snapshot_rename flags 3 0x7fb5ff451938 2022-03-29T17:49:53.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.snapshot_trash_add flags 3 0x7fb5ff45082f 2022-03-29T17:49:53.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_all_features flags 1 0x7fb5ff4270bc 2022-03-29T17:49:53.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_parent flags 1 0x7fb5ff44ed9f 2022-03-29T17:49:53.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.set_parent flags 3 0x7fb5ff441007 2022-03-29T17:49:53.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.remove_parent flags 3 0x7fb5ff453a52 2022-03-29T17:49:53.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.parent_get flags 1 0x7fb5ff44f91f 2022-03-29T17:49:53.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.parent_overlap_get flags 1 0x7fb5ff457270 2022-03-29T17:49:53.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.parent_attach flags 3 0x7fb5ff437489 2022-03-29T17:49:53.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.parent_detach flags 3 0x7fb5ff453a72 2022-03-29T17:49:53.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.set_protection_status flags 3 0x7fb5ff4567f9 2022-03-29T17:49:53.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_protection_status flags 1 0x7fb5ff455655 2022-03-29T17:49:53.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_stripe_unit_count flags 1 0x7fb5ff434140 2022-03-29T17:49:53.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.set_stripe_unit_count flags 3 0x7fb5ff4335c9 2022-03-29T17:49:53.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_create_timestamp flags 1 0x7fb5ff430e65 2022-03-29T17:49:53.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_access_timestamp flags 1 0x7fb5ff4310d3 2022-03-29T17:49:53.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_modify_timestamp flags 1 0x7fb5ff431341 2022-03-29T17:49:53.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.get_flags flags 1 0x7fb5ff457842 2022-03-29T17:49:53.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.set_flags flags 3 0x7fb5ff453a92 2022-03-29T17:49:53.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.op_features_get flags 1 0x7fb5ff43044f 2022-03-29T17:49:53.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.op_features_set flags 3 0x7fb5ff434a28 2022-03-29T17:49:53.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.metadata_list flags 1 0x7fb5ff445d62 2022-03-29T17:49:53.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.metadata_set flags 3 0x7fb5ff44e5a7 2022-03-29T17:49:53.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.metadata_remove flags 3 0x7fb5ff42de87 2022-03-29T17:49:53.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.metadata_get flags 1 0x7fb5ff42e065 2022-03-29T17:49:53.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.snapshot_get_limit flags 1 0x7fb5ff430731 2022-03-29T17:49:53.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.snapshot_set_limit flags 3 0x7fb5ff433079 2022-03-29T17:49:53.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.child_attach flags 3 0x7fb5ff45473a 2022-03-29T17:49:53.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.child_detach flags 3 0x7fb5ff4560fc 2022-03-29T17:49:53.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.children_list flags 1 0x7fb5ff450329 2022-03-29T17:49:53.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.migration_set flags 3 0x7fb5ff435b72 2022-03-29T17:49:53.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,395 [shard 0] osd - register_cxx_method rbd.migration_set_state flags 3 0x7fb5ff435dc9 2022-03-29T17:49:53.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.migration_get flags 1 0x7fb5ff432075 2022-03-29T17:49:53.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.migration_remove flags 3 0x7fb5ff434125 2022-03-29T17:49:53.626 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.set_modify_timestamp flags 3 0x7fb5ff4309aa 2022-03-29T17:49:53.626 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.set_access_timestamp flags 3 0x7fb5ff4308a5 2022-03-29T17:49:53.626 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.add_child flags 3 0x7fb5ff458af5 2022-03-29T17:49:53.626 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.remove_child flags 3 0x7fb5ff459052 2022-03-29T17:49:53.627 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.get_children flags 1 0x7fb5ff459633 2022-03-29T17:49:53.627 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.get_id flags 1 0x7fb5ff42948d 2022-03-29T17:49:53.627 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.set_id flags 3 0x7fb5ff4296f0 2022-03-29T17:49:53.627 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.dir_get_id flags 1 0x7fb5ff42cc9d 2022-03-29T17:49:53.627 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.dir_get_name flags 1 0x7fb5ff42d59c 2022-03-29T17:49:53.628 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.dir_list flags 1 0x7fb5ff4435f8 2022-03-29T17:49:53.628 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.dir_add_image flags 3 0x7fb5ff447a41 2022-03-29T17:49:53.628 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.dir_remove_image flags 3 0x7fb5ff42d41b 2022-03-29T17:49:53.628 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.dir_rename_image flags 3 0x7fb5ff44780d 2022-03-29T17:49:53.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.dir_state_assert flags 1 0x7fb5ff43033d 2022-03-29T17:49:53.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.dir_state_set flags 3 0x7fb5ff432df0 2022-03-29T17:49:53.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.object_map_load flags 1 0x7fb5ff447dee 2022-03-29T17:49:53.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.object_map_save flags 3 0x7fb5ff4482fc 2022-03-29T17:49:53.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.object_map_resize flags 3 0x7fb5ff449a9b 2022-03-29T17:49:53.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.object_map_update flags 3 0x7fb5ff448bce 2022-03-29T17:49:53.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.object_map_snap_add flags 3 0x7fb5ff447ec6 2022-03-29T17:49:53.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.object_map_snap_remove flags 3 0x7fb5ff4484bf 2022-03-29T17:49:53.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.snap_list flags 1 0x7fb5ff42ff91 2022-03-29T17:49:53.631 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.snap_add flags 3 0x7fb5ff43268e 2022-03-29T17:49:53.631 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.snap_remove flags 3 0x7fb5ff429b41 2022-03-29T17:49:53.631 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.snap_rename flags 3 0x7fb5ff42a2a2 2022-03-29T17:49:53.631 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.mirror_uuid_get flags 1 0x7fb5ff42fcef 2022-03-29T17:49:53.632 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.mirror_uuid_set flags 3 0x7fb5ff43c253 2022-03-29T17:49:53.632 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.mirror_mode_get flags 1 0x7fb5ff42ac4b 2022-03-29T17:49:53.632 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.mirror_mode_set flags 3 0x7fb5ff43fba6 2022-03-29T17:49:53.632 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.mirror_peer_ping flags 3 0x7fb5ff44e3fb 2022-03-29T17:49:53.632 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.mirror_peer_list flags 1 0x7fb5ff440035 2022-03-29T17:49:53.633 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.mirror_peer_add flags 3 0x7fb5ff4412de 2022-03-29T17:49:53.633 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,396 [shard 0] osd - register_cxx_method rbd.mirror_peer_remove flags 3 0x7fb5ff43813d 2022-03-29T17:49:53.634 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,396 [shard 0] monc - responding with 36 bytes 2022-03-29T17:49:53.634 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,397 [shard 0] monc - found mon.noname-a 2022-03-29T17:49:53.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] monc - reopen_session mon connection attempts complete 2022-03-29T17:49:53.635 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,397 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2022-03-29T17:49:53.635 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,397 [shard 0] monc - waiting 2022-03-29T17:49:53.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_client flags 3 0x7fb5ff437b04 2022-03-29T17:49:53.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_cluster flags 3 0x7fb5ff44018f 2022-03-29T17:49:53.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_peer_set_direction flags 3 0x7fb5ff437ddb 2022-03-29T17:49:53.656 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_list flags 1 0x7fb5ff442cae 2022-03-29T17:49:53.656 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_get_image_id flags 1 0x7fb5ff438e8c 2022-03-29T17:49:53.656 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_get flags 1 0x7fb5ff43849a 2022-03-29T17:49:53.656 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_set flags 3 0x7fb5ff438d0f 2022-03-29T17:49:53.656 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_remove flags 3 0x7fb5ff43ef25 2022-03-29T17:49:53.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_status_set flags 11 0x7fb5ff43e7da 2022-03-29T17:49:53.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove flags 3 0x7fb5ff43f02f 2022-03-29T17:49:53.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get flags 1 0x7fb5ff440e5f 2022-03-29T17:49:53.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_status_list flags 1 0x7fb5ff44ad7f 2022-03-29T17:49:53.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_status_get_summary flags 1 0x7fb5ff44b570 2022-03-29T17:49:53.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_status_remove_down flags 3 0x7fb5ff43df42 2022-03-29T17:49:53.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_get flags 1 0x7fb5ff43e2cd 2022-03-29T17:49:53.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_instance_list flags 1 0x7fb5ff44c04d 2022-03-29T17:49:53.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_instances_list flags 1 0x7fb5ff43c851 2022-03-29T17:49:53.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_instances_add flags 11 0x7fb5ff4394de 2022-03-29T17:49:53.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_instances_remove flags 3 0x7fb5ff4396ef 2022-03-29T17:49:53.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_map_list flags 1 0x7fb5ff43c05e 2022-03-29T17:49:53.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_map_update flags 2 0x7fb5ff439841 2022-03-29T17:49:53.660 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_map_remove flags 2 0x7fb5ff439bbd 2022-03-29T17:49:53.660 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_unlink_peer flags 3 0x7fb5ff45ab3c 2022-03-29T17:49:53.660 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.mirror_image_snapshot_set_copy_progress flags 3 0x7fb5ff459e49 2022-03-29T17:49:53.660 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,397 [shard 0] osd - register_cxx_method rbd.group_dir_list flags 1 0x7fb5ff442196 2022-03-29T17:49:53.660 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_dir_add flags 3 0x7fb5ff446ca7 2022-03-29T17:49:53.661 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_dir_remove flags 3 0x7fb5ff42dcc2 2022-03-29T17:49:53.661 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_dir_rename flags 3 0x7fb5ff446ec2 2022-03-29T17:49:53.661 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_image_remove flags 3 0x7fb5ff42732a 2022-03-29T17:49:53.661 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_image_list flags 1 0x7fb5ff43cc9a 2022-03-29T17:49:53.662 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_image_set flags 3 0x7fb5ff42acb3 2022-03-29T17:49:53.662 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.image_group_add flags 3 0x7fb5ff434b03 2022-03-29T17:49:53.662 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.image_group_remove flags 3 0x7fb5ff434f46 2022-03-29T17:49:53.662 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.image_group_get flags 1 0x7fb5ff430c65 2022-03-29T17:49:53.663 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_snap_set flags 3 0x7fb5ff44cae6 2022-03-29T17:49:53.663 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_snap_remove flags 3 0x7fb5ff42eb76 2022-03-29T17:49:53.663 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_snap_get_by_id flags 1 0x7fb5ff42ed09 2022-03-29T17:49:53.663 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.group_snap_list flags 1 0x7fb5ff441c44 2022-03-29T17:49:53.663 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.trash_add flags 3 0x7fb5ff44579b 2022-03-29T17:49:53.664 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.trash_remove flags 3 0x7fb5ff439de3 2022-03-29T17:49:53.664 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.trash_list flags 1 0x7fb5ff44cece 2022-03-29T17:49:53.664 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.trash_get flags 1 0x7fb5ff43a0e8 2022-03-29T17:49:53.664 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.trash_state_set flags 3 0x7fb5ff43a36e 2022-03-29T17:49:53.665 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.namespace_add flags 3 0x7fb5ff43a7a6 2022-03-29T17:49:53.665 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.namespace_remove flags 3 0x7fb5ff43a9d1 2022-03-29T17:49:53.665 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.namespace_list flags 1 0x7fb5ff43abba 2022-03-29T17:49:53.665 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.copyup flags 3 0x7fb5ff42716b 2022-03-29T17:49:53.665 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.sparse_copyup flags 3 0x7fb5ff457e66 2022-03-29T17:49:53.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.assert_snapc_seq flags 3 0x7fb5ff42721b 2022-03-29T17:49:53.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method rbd.sparsify flags 3 0x7fb5ff42b074 2022-03-29T17:49:53.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - _load_class rbd success 2022-03-29T17:49:53.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - open_all_classes found queue 2022-03-29T17:49:53.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - _get_class adding new class name queue 0x612000026900 2022-03-29T17:49:53.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - _load_class queue from /usr/lib64/rados-classes/libcls_queue.so 2022-03-29T17:49:53.667 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,398 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/queue/cls_queue.cc:125: Loaded queue class! 2022-03-29T17:49:53.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_class queue status 3 2022-03-29T17:49:53.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method queue.queue_init flags 3 0x7fb5ff1eb256 2022-03-29T17:49:53.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method queue.queue_get_capacity flags 1 0x7fb5ff1ea9e9 2022-03-29T17:49:53.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method queue.queue_enqueue flags 3 0x7fb5ff1eb52e 2022-03-29T17:49:53.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method queue.queue_list_entries flags 1 0x7fb5ff1eadaf 2022-03-29T17:49:53.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method queue.queue_remove_entries flags 3 0x7fb5ff1eab81 2022-03-29T17:49:53.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - _load_class queue success 2022-03-29T17:49:53.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - open_all_classes found user 2022-03-29T17:49:53.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - _get_class adding new class name user 0x612000026c00 2022-03-29T17:49:53.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - _load_class user from /usr/lib64/rados-classes/libcls_user.so 2022-03-29T17:49:53.669 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,398 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/user/cls_user.cc:506: Loaded user class! 2022-03-29T17:49:53.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_class user status 3 2022-03-29T17:49:53.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method user.set_buckets_info flags 3 0x7fb5fefd1c4b 2022-03-29T17:49:53.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method user.complete_stats_sync flags 3 0x7fb5fefd19a3 2022-03-29T17:49:53.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method user.remove_bucket flags 3 0x7fb5fefd11be 2022-03-29T17:49:53.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method user.list_buckets flags 1 0x7fb5fefd2733 2022-03-29T17:49:53.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method user.get_header flags 1 0x7fb5fefd0fc7 2022-03-29T17:49:53.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method user.reset_user_stats flags 3 0x7fb5fefd054c 2022-03-29T17:49:53.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - register_cxx_method user.reset_user_stats2 flags 3 0x7fb5fefd319b 2022-03-29T17:49:53.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - _load_class user success 2022-03-29T17:49:53.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,398 [shard 0] osd - open_all_classes found cas 2022-03-29T17:49:53.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _get_class adding new class name cas 0x612000027080 2022-03-29T17:49:53.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _load_class cas from /usr/lib64/rados-classes/libcls_cas.so 2022-03-29T17:49:53.672 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,399 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cas/cls_cas.cc:211: Loaded cas class! 2022-03-29T17:49:53.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_class cas status 3 2022-03-29T17:49:53.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method cas.chunk_create_or_get_ref flags 3 0x7fb5fedbab38 2022-03-29T17:49:53.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method cas.chunk_get_ref flags 3 0x7fb5fedba848 2022-03-29T17:49:53.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method cas.chunk_put_ref flags 3 0x7fb5fedba4c4 2022-03-29T17:49:53.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method cas.references_chunk flags 1 0x7fb5fedba1fc 2022-03-29T17:49:53.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _load_class cas success 2022-03-29T17:49:53.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - open_all_classes found fifo 2022-03-29T17:49:53.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _get_class adding new class name fifo 0x612000027380 2022-03-29T17:49:53.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _load_class fifo from /usr/lib64/rados-classes/libcls_fifo.so 2022-03-29T17:49:53.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_class fifo status 3 2022-03-29T17:49:53.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method fifo.create_meta flags 3 0x7fb5feb7bdee 2022-03-29T17:49:53.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method fifo.get_meta flags 1 0x7fb5feb7d0b2 2022-03-29T17:49:53.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method fifo.update_meta flags 3 0x7fb5feb7d57b 2022-03-29T17:49:53.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method fifo.init_part flags 3 0x7fb5feb7acf2 2022-03-29T17:49:53.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method fifo.push_part flags 3 0x7fb5feb7b3e5 2022-03-29T17:49:53.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method fifo.trim_part flags 3 0x7fb5feb7a782 2022-03-29T17:49:53.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method fifo.part_list flags 1 0x7fb5feb79ed9 2022-03-29T17:49:53.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method fifo.get_part_info flags 1 0x7fb5feb7977e 2022-03-29T17:49:53.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _load_class fifo success 2022-03-29T17:49:53.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - open_all_classes found cmpomap 2022-03-29T17:49:53.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _get_class adding new class name cmpomap 0x612000027680 2022-03-29T17:49:53.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _load_class cmpomap from /usr/lib64/rados-classes/libcls_cmpomap.so 2022-03-29T17:49:53.677 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,399 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/cmpomap/server.cc:287: Loaded cmpomap class! 2022-03-29T17:49:53.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_class cmpomap status 3 2022-03-29T17:49:53.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method cmpomap.cmp_vals flags 1 0x7fb5fe95e2e7 2022-03-29T17:49:53.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method cmpomap.cmp_set_vals flags 3 0x7fb5fe95d70e 2022-03-29T17:49:53.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - register_cxx_method cmpomap.cmp_rm_keys flags 3 0x7fb5fe95d136 2022-03-29T17:49:53.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _load_class cmpomap success 2022-03-29T17:49:53.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - open_all_classes found hello 2022-03-29T17:49:53.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _get_class adding new class name hello 0x612000027b00 2022-03-29T17:49:53.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,399 [shard 0] osd - _load_class hello from /usr/lib64/rados-classes/libcls_hello.so 2022-03-29T17:49:53.679 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,400 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/hello/cls_hello.cc:316: loading cls_hello 2022-03-29T17:49:53.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_class hello status 3 2022-03-29T17:49:53.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.say_hello flags 1 0x7fb5fe74dbf6 2022-03-29T17:49:53.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.record_hello flags 10 0x7fb5fe74dea9 2022-03-29T17:49:53.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.write_return_data flags 2 0x7fb5fe74dd7f 2022-03-29T17:49:53.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.writes_dont_return_data flags 2 0x7fb5fe74dd7f 2022-03-29T17:49:53.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.write_too_much_return_data flags 2 0x7fb5fe74dc6d 2022-03-29T17:49:53.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.replay flags 1 0x7fb5fe74dbce 2022-03-29T17:49:53.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.turn_it_to_11 flags 11 0x7fb5fe74e808 2022-03-29T17:49:53.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.bad_reader flags 2 0x7fb5fe74dbaf 2022-03-29T17:49:53.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method hello.bad_writer flags 1 0x7fb5fe74db9d 2022-03-29T17:49:53.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _load_class hello success 2022-03-29T17:49:53.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - open_all_classes found log 2022-03-29T17:49:53.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _get_class adding new class name log 0x612000027e00 2022-03-29T17:49:53.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _load_class log from /usr/lib64/rados-classes/libcls_log.so 2022-03-29T17:49:53.683 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,400 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/log/cls_log.cc:305: Loaded log class! 2022-03-29T17:49:53.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_class log status 3 2022-03-29T17:49:53.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method log.add flags 3 0x7fb5fe53cf1f 2022-03-29T17:49:53.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method log.list flags 1 0x7fb5fe53da5d 2022-03-29T17:49:53.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method log.trim flags 3 0x7fb5fe53d2cf 2022-03-29T17:49:53.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method log.info flags 1 0x7fb5fe53caf7 2022-03-29T17:49:53.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _load_class log success 2022-03-29T17:49:53.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - open_all_classes found otp 2022-03-29T17:49:53.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _get_class adding new class name otp 0x612000028280 2022-03-29T17:49:53.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _load_class otp from /usr/lib64/rados-classes/libcls_otp.so 2022-03-29T17:49:53.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_class otp status 3 2022-03-29T17:49:53.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method otp.otp_set flags 3 0x7fb5fe32937f 2022-03-29T17:49:53.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method otp.otp_get flags 1 0x7fb5fe32898f 2022-03-29T17:49:53.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method otp.otp_check flags 3 0x7fb5fe327fc3 2022-03-29T17:49:53.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method otp.otp_get_result flags 1 0x7fb5fe327902 2022-03-29T17:49:53.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method otp.otp_remove flags 3 0x7fb5fe328700 2022-03-29T17:49:53.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method otp.get_current_time flags 1 0x7fb5fe3275f0 2022-03-29T17:49:53.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _load_class otp success 2022-03-29T17:49:53.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - open_all_classes found sdk 2022-03-29T17:49:53.687 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,400 [shard 0] osd - _get_class not permitted to load sdk 2022-03-29T17:49:53.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - open_all_classes found timeindex 2022-03-29T17:49:53.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _get_class adding new class name timeindex 0x612000028400 2022-03-29T17:49:53.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - _load_class timeindex from /usr/lib64/rados-classes/libcls_timeindex.so 2022-03-29T17:49:53.688 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,400 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/timeindex/cls_timeindex.cc:247: Loaded timeindex class! 2022-03-29T17:49:53.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_class timeindex status 3 2022-03-29T17:49:53.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method timeindex.add flags 3 0x7fb5fdefa894 2022-03-29T17:49:53.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method timeindex.list flags 1 0x7fb5fdefac23 2022-03-29T17:49:53.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,400 [shard 0] osd - register_cxx_method timeindex.trim flags 3 0x7fb5fdefa43d 2022-03-29T17:49:53.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _load_class timeindex success 2022-03-29T17:49:53.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - open_all_classes found lock 2022-03-29T17:49:53.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _get_class adding new class name lock 0x612000028880 2022-03-29T17:49:53.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _load_class lock from /usr/lib64/rados-classes/libcls_lock.so 2022-03-29T17:49:53.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_class lock status 3 2022-03-29T17:49:53.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method lock.lock flags 11 0x7fb5fdce74af 2022-03-29T17:49:53.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method lock.unlock flags 11 0x7fb5fdce4e27 2022-03-29T17:49:53.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method lock.break_lock flags 3 0x7fb5fdce4ccd 2022-03-29T17:49:53.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method lock.get_info flags 1 0x7fb5fdce5a33 2022-03-29T17:49:53.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method lock.list_locks flags 1 0x7fb5fdce326b 2022-03-29T17:49:53.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method lock.assert_locked flags 9 0x7fb5fdce43bd 2022-03-29T17:49:53.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method lock.set_cookie flags 11 0x7fb5fdce4ff4 2022-03-29T17:49:53.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _load_class lock success 2022-03-29T17:49:53.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - open_all_classes found 2pc_queue 2022-03-29T17:49:53.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _get_class adding new class name 2pc_queue 0x612000028a00 2022-03-29T17:49:53.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _load_class 2pc_queue from /usr/lib64/rados-classes/libcls_2pc_queue.so 2022-03-29T17:49:53.692 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,401 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/2pc_queue/cls_2pc_queue.cc:575: Loaded 2pc queue class! 2022-03-29T17:49:53.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_class 2pc_queue status 3 2022-03-29T17:49:53.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_init flags 3 0x7fb5fdaa66a3 2022-03-29T17:49:53.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_get_capacity flags 1 0x7fb5fdaa5c19 2022-03-29T17:49:53.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_reserve flags 3 0x7fb5fdaa9504 2022-03-29T17:49:53.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_commit flags 3 0x7fb5fdaa6b0d 2022-03-29T17:49:53.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_abort flags 3 0x7fb5fdaa828f 2022-03-29T17:49:53.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_reservations flags 1 0x7fb5fdaa8c8c 2022-03-29T17:49:53.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_list_entries flags 1 0x7fb5fdaa6076 2022-03-29T17:49:53.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_remove_entries flags 3 0x7fb5fdaa5e39 2022-03-29T17:49:53.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - register_cxx_method 2pc_queue.2pc_queue_expire_reservations flags 3 0x7fb5fdaa77d0 2022-03-29T17:49:53.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _load_class 2pc_queue success 2022-03-29T17:49:53.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - open_all_classes found refcount 2022-03-29T17:49:53.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _get_class adding new class name refcount 0x612000028d00 2022-03-29T17:49:53.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,401 [shard 0] osd - _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so 2022-03-29T17:49:53.696 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,403 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/refcount/cls_refcount.cc:199: Loaded refcount class! 2022-03-29T17:49:53.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_class refcount status 3 2022-03-29T17:49:53.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method refcount.get flags 3 0x7fb5fd7dec89 2022-03-29T17:49:53.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method refcount.put flags 3 0x7fb5fd7de94c 2022-03-29T17:49:53.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method refcount.set flags 3 0x7fb5fd7de0c1 2022-03-29T17:49:53.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method refcount.read flags 1 0x7fb5fd7de602 2022-03-29T17:49:53.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - _load_class refcount success 2022-03-29T17:49:53.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - open_all_classes found version 2022-03-29T17:49:53.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - _get_class adding new class name version 0x612000029000 2022-03-29T17:49:53.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - _load_class version from /usr/lib64/rados-classes/libcls_version.so 2022-03-29T17:49:53.698 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,403 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/version/cls_version.cc:218: Loaded version class! 2022-03-29T17:49:53.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_class version status 3 2022-03-29T17:49:53.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method version.set flags 3 0x7fb5fd52d3d7 2022-03-29T17:49:53.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method version.inc flags 3 0x7fb5fd52d6e1 2022-03-29T17:49:53.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method version.inc_conds flags 3 0x7fb5fd52d6e1 2022-03-29T17:49:53.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method version.read flags 1 0x7fb5fd52d21c 2022-03-29T17:49:53.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method version.check_conds flags 1 0x7fb5fd52d4f9 2022-03-29T17:49:53.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - _load_class version success 2022-03-29T17:49:53.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - open_all_classes found rgw_gc 2022-03-29T17:49:53.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - _get_class adding new class name rgw_gc 0x612000029480 2022-03-29T17:49:53.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - _load_class rgw_gc from /usr/lib64/rados-classes/libcls_rgw_gc.so 2022-03-29T17:49:53.701 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,403 [shard 0] objclass - /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/cls/rgw_gc/cls_rgw_gc.cc:539: Loaded rgw gc class! 2022-03-29T17:49:53.701 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_class rgw_gc status 3 2022-03-29T17:49:53.701 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_init flags 3 0x7fb5fd2f0a31 2022-03-29T17:49:53.701 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_enqueue flags 3 0x7fb5fd2f0130 2022-03-29T17:49:53.701 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_list_entries flags 1 0x7fb5fd2f22eb 2022-03-29T17:49:53.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_remove_entries flags 3 0x7fb5fd2f0d97 2022-03-29T17:49:53.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - register_cxx_method rgw_gc.rgw_gc_queue_update_entry flags 3 0x7fb5fd2f3e0e 2022-03-29T17:49:53.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,403 [shard 0] osd - _load_class rgw_gc success 2022-03-29T17:49:53.702 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,403 [shard 0] osd - OSD: nonce is 35577 2022-03-29T17:49:53.703 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding auth protocol: cephx 2022-03-29T17:49:53.703 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding auth protocol: cephx 2022-03-29T17:49:53.703 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding auth protocol: cephx 2022-03-29T17:49:53.703 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding con mode: crc 2022-03-29T17:49:53.703 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding con mode: crc 2022-03-29T17:49:53.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding con mode: crc 2022-03-29T17:49:53.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding con mode: crc 2022-03-29T17:49:53.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding con mode: crc 2022-03-29T17:49:53.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - AuthRegistry(0x7fb5fd2e2b60) adding con mode: crc 2022-03-29T17:49:53.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,404 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-0/keyring 2022-03-29T17:49:53.705 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,414 [shard 0] monc - build_monmap: for_mkfs=false 2022-03-29T17:49:53.705 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,414 [shard 0] monc - reopen_session to mon.-1 2022-03-29T17:49:53.705 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,414 [shard 0] monc - mon.0 does not have an addr compatible with me 2022-03-29T17:49:53.705 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,414 [shard 0] monc - connecting to mon.1 2022-03-29T17:49:53.706 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,414 [shard 0] ms - [client.?(temp_mon_client) - >> mon.? v2:172.21.15.138:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:3300/0, peer_name=mon.?, cc=13073382650400683706 policy(lossy=false, server=false, standby=false, resetcheck=true) 2022-03-29T17:49:53.706 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,414 [shard 0] ms - [client.?(temp_mon_client) - >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:49:53.706 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,415 [shard 0] ms - [client.?(temp_mon_client) -@52796 >> mon.? v2:172.21.15.138:3300/0] peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:52796/0 (socket says 172.21.15.138:52796) 2022-03-29T17:49:53.706 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,415 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35577@52796 >> mon.? v2:172.21.15.138:3300/0] learned myaddr=172.21.15.138:0/35577 (unbound) 2022-03-29T17:49:53.706 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,415 [shard 0] monc - get_auth_request(con=[client.?(temp_mon_client) 172.21.15.138:0/35577@52796 >> mon.? v2:172.21.15.138:3300/0], auth_method=0) 2022-03-29T17:49:53.707 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,415 [shard 0] monc - method 2 preferred_modes {1} 2022-03-29T17:49:53.708 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,422 [shard 0] monc - responding with 36 bytes 2022-03-29T17:49:53.708 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,423 [shard 0] monc - found mon.noname-a 2022-03-29T17:49:53.708 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,423 [shard 0] monc - reopen_session mon connection attempts complete 2022-03-29T17:49:53.708 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,423 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2022-03-29T17:49:53.709 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,423 [shard 0] monc - waiting 2022-03-29T17:49:53.710 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,437 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35578@57921 >> mon.0 v2:172.21.15.138:3300/0] UPDATE Policy(lossy=true) from server flags 2022-03-29T17:49:53.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,437 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35578@57921 >> mon.0 v2:172.21.15.138:3300/0] connected: gs=1, pgs=22, cs=0, client_cookie=216034701559232259, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2022-03-29T17:49:53.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,438 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2022-03-29T17:49:53.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,438 [shard 0] monc - handle_monmap: renewing tickets 2022-03-29T17:49:53.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,438 [shard 0] monc - renew_rotating_keyring called too often 2022-03-29T17:49:53.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,438 [shard 0] monc - handle_mon_map: renewed tickets 2022-03-29T17:49:53.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,438 [shard 0] monc - handle_auth_reply mon 172.21.15.138:0/35578 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,438 [shard 0] monc - handle_auth_reply 2022-03-29T17:49:53.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,438 [shard 0] monc - do_auth_single: mon 172.21.15.138:0/35578 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,438 [shard 0] monc - set_mon_vals no callback set 2022-03-29T17:49:53.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,441 [shard 0] monc - stop 2022-03-29T17:49:53.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,441 [shard 0] monc - close 2022-03-29T17:49:53.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,441 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35578@57921 >> mon.0 v2:172.21.15.138:3300/0] closing: reset no, replace no 2022-03-29T17:49:53.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,442 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35578@57921 >> mon.0 v2:172.21.15.138:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:53.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,442 [shard 0] osd - start 2022-03-29T17:49:53.714 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,455 [shard 0] monc - responding with 36 bytes 2022-03-29T17:49:53.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,456 [shard 0] monc - found mon.noname-a 2022-03-29T17:49:53.715 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,456 [shard 0] monc - reopen_session mon connection attempts complete 2022-03-29T17:49:53.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,456 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2022-03-29T17:49:53.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,456 [shard 0] monc - waiting 2022-03-29T17:49:53.717 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,463 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35579@57488 >> mon.0 v2:172.21.15.138:3300/0] UPDATE Policy(lossy=true) from server flags 2022-03-29T17:49:53.718 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,463 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35579@57488 >> mon.0 v2:172.21.15.138:3300/0] connected: gs=1, pgs=23, cs=0, client_cookie=1072012829979169547, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2022-03-29T17:49:53.718 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,464 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2022-03-29T17:49:53.718 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,464 [shard 0] monc - handle_monmap: renewing tickets 2022-03-29T17:49:53.718 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,464 [shard 0] monc - renew_rotating_keyring called too often 2022-03-29T17:49:53.719 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,464 [shard 0] monc - handle_mon_map: renewed tickets 2022-03-29T17:49:53.719 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,464 [shard 0] monc - handle_auth_reply mon 172.21.15.138:0/35579 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.719 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,464 [shard 0] monc - handle_auth_reply 2022-03-29T17:49:53.719 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,464 [shard 0] monc - do_auth_single: mon 172.21.15.138:0/35579 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,464 [shard 0] monc - set_mon_vals no callback set 2022-03-29T17:49:53.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,468 [shard 0] monc - stop 2022-03-29T17:49:53.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,468 [shard 0] monc - close 2022-03-29T17:49:53.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,468 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35579@57488 >> mon.0 v2:172.21.15.138:3300/0] closing: reset no, replace no 2022-03-29T17:49:53.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,468 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35579@57488 >> mon.0 v2:172.21.15.138:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:53.721 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,468 [shard 0] osd - start 2022-03-29T17:49:53.721 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:53.490+0000 7fb027daaf40 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-03-29T17:49:53.723 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,497 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35577@52796 >> mon.0 v2:172.21.15.138:3300/0] UPDATE Policy(lossy=true) from server flags 2022-03-29T17:49:53.723 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,497 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35577@52796 >> mon.0 v2:172.21.15.138:3300/0] connected: gs=1, pgs=24, cs=0, client_cookie=13073382650400683706, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2022-03-29T17:49:53.724 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,498 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2022-03-29T17:49:53.724 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,498 [shard 0] monc - handle_monmap: renewing tickets 2022-03-29T17:49:53.724 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,498 [shard 0] monc - renew_rotating_keyring called too often 2022-03-29T17:49:53.724 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,498 [shard 0] monc - handle_mon_map: renewed tickets 2022-03-29T17:49:53.725 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,498 [shard 0] monc - handle_auth_reply mon 172.21.15.138:0/35577 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.725 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,498 [shard 0] monc - handle_auth_reply 2022-03-29T17:49:53.725 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,498 [shard 0] monc - do_auth_single: mon 172.21.15.138:0/35577 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.725 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,499 [shard 0] monc - set_mon_vals no callback set 2022-03-29T17:49:53.725 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,503 [shard 0] monc - stop 2022-03-29T17:49:53.726 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,503 [shard 0] monc - close 2022-03-29T17:49:53.726 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,503 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35577@52796 >> mon.0 v2:172.21.15.138:3300/0] closing: reset no, replace no 2022-03-29T17:49:53.726 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,503 [shard 0] ms - [client.?(temp_mon_client) 172.21.15.138:0/35577@52796 >> mon.0 v2:172.21.15.138:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:53.726 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,503 [shard 0] osd - start 2022-03-29T17:49:53.727 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:53.568+0000 7fb027daaf40 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-03-29T17:49:53.727 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:53.672+0000 7fb027daaf40 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-03-29T17:49:53.747 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:53.744+0000 7fb027daaf40 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-03-29T17:49:53.777 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,775 [shard 0] osd - ignoring unrecognized collection: meta 2022-03-29T17:49:53.777 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,776 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:53.778 INFO:tasks.ceph.osd.1.smithi138.stderr:ERROR 2022-03-29 17:49:53,776 [shard 0] none - Falling back to public interface 2022-03-29T17:49:53.778 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,776 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:53.780 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] ms - [osd.1(client) v2:0.0.0.0:6800/35578] try_bind: done 2022-03-29T17:49:53.780 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] ms - [osd.1(client) v2:0.0.0.0:6800/35578] try_bind: done 2022-03-29T17:49:53.780 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] ms - [osd.1(cluster) v2:0.0.0.0:6801/35578] try_bind: done 2022-03-29T17:49:53.781 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] ms - [osd.1(cluster) v2:0.0.0.0:6801/35578] try_bind: done 2022-03-29T17:49:53.781 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.781 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.781 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.782 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.782 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.782 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.782 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.782 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.783 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,777 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.783 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,778 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-1/keyring 2022-03-29T17:49:53.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,780 [shard 0] monc - build_monmap: for_mkfs=false 2022-03-29T17:49:53.785 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,781 [shard 0] monc - reopen_session to mon.-1 2022-03-29T17:49:53.785 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,781 [shard 0] monc - connecting to mon.1 2022-03-29T17:49:53.785 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,781 [shard 0] ms - [osd.1(client) v2:0.0.0.0:6800/35578 >> mon.? v2:172.21.15.138:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:3300/0, peer_name=mon.?, cc=8058926782284612243 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:53.785 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,781 [shard 0] ms - [osd.1(client) v2:0.0.0.0:6800/35578 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:49:53.786 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:53,781 [shard 0] monc - mon.0 does not have an addr compatible with me 2022-03-29T17:49:53.786 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,782 [shard 0] ms - [osd.1(client) v2:0.0.0.0:6800/35578@58384 >> mon.? v2:172.21.15.138:3300/0] peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:58384/0 (socket says 172.21.15.138:58384) 2022-03-29T17:49:53.786 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,782 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@58384 >> mon.? v2:172.21.15.138:3300/0] learned myaddr=v2:172.21.15.138:6800/35578 (blank IP) 2022-03-29T17:49:53.786 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,782 [shard 0] monc - get_auth_request(con=[osd.1(client) v2:172.21.15.138:6800/35578@58384 >> mon.? v2:172.21.15.138:3300/0], auth_method=0) 2022-03-29T17:49:53.787 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,782 [shard 0] monc - method 2 preferred_modes {1} 2022-03-29T17:49:53.824 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,822 [shard 0] monc - responding with 36 bytes 2022-03-29T17:49:53.825 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,823 [shard 0] monc - found mon.noname-a 2022-03-29T17:49:53.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:53,823 [shard 0] monc - reopen_session mon connection attempts complete 2022-03-29T17:49:53.825 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,823 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2022-03-29T17:49:53.826 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,823 [shard 0] monc - waiting 2022-03-29T17:49:53.836 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,834 [shard 0] osd - ignoring unrecognized collection: meta 2022-03-29T17:49:53.837 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,835 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:53.837 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:49:53,835 [shard 0] none - Falling back to public interface 2022-03-29T17:49:53.837 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,835 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:53.839 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,836 [shard 0] ms - [osd.2(cluster) v2:0.0.0.0:6802/35579] try_bind: done 2022-03-29T17:49:53.839 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,836 [shard 0] ms - [osd.2(cluster) v2:0.0.0.0:6802/35579] try_bind: done 2022-03-29T17:49:53.840 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] ms - [osd.2(client) v2:0.0.0.0:6803/35579] try_bind: done 2022-03-29T17:49:53.840 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] ms - [osd.2(client) v2:0.0.0.0:6803/35579] try_bind: done 2022-03-29T17:49:53.840 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.840 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.840 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.841 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.841 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.841 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.841 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.842 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.842 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,837 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.842 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,838 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring 2022-03-29T17:49:53.849 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,846 [shard 0] monc - build_monmap: for_mkfs=false 2022-03-29T17:49:53.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,846 [shard 0] monc - reopen_session to mon.-1 2022-03-29T17:49:53.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,847 [shard 0] monc - connecting to mon.1 2022-03-29T17:49:53.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,847 [shard 0] ms - [osd.2(client) v2:0.0.0.0:6803/35579 >> mon.? v2:172.21.15.138:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:3300/0, peer_name=mon.?, cc=12217060769435700643 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:53.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,847 [shard 0] ms - [osd.2(client) v2:0.0.0.0:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:49:53.850 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:53,847 [shard 0] monc - mon.0 does not have an addr compatible with me 2022-03-29T17:49:53.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,847 [shard 0] ms - [osd.2(client) v2:0.0.0.0:6803/35579@56603 >> mon.? v2:172.21.15.138:3300/0] peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:56603/0 (socket says 172.21.15.138:56603) 2022-03-29T17:49:53.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,847 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@56603 >> mon.? v2:172.21.15.138:3300/0] learned myaddr=v2:172.21.15.138:6803/35579 (blank IP) 2022-03-29T17:49:53.851 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,847 [shard 0] monc - get_auth_request(con=[osd.2(client) v2:172.21.15.138:6803/35579@56603 >> mon.? v2:172.21.15.138:3300/0], auth_method=0) 2022-03-29T17:49:53.851 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,847 [shard 0] monc - method 2 preferred_modes {1} 2022-03-29T17:49:53.852 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,851 [shard 0] osd - ignoring unrecognized collection: meta 2022-03-29T17:49:53.853 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,851 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:53.853 INFO:tasks.ceph.osd.0.smithi138.stderr:ERROR 2022-03-29 17:49:53,851 [shard 0] none - Falling back to public interface 2022-03-29T17:49:53.853 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,851 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:53.855 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6804/35577] try_bind: done 2022-03-29T17:49:53.855 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6804/35577] try_bind: done 2022-03-29T17:49:53.855 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6805/35577] try_bind: done 2022-03-29T17:49:53.856 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6805/35577] try_bind: done 2022-03-29T17:49:53.856 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.856 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.856 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding auth protocol: cephx 2022-03-29T17:49:53.857 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.857 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.857 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.857 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.857 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.858 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,852 [shard 0] auth - AuthRegistry(0x61b000001230) adding con mode: crc 2022-03-29T17:49:53.858 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,853 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-0/keyring 2022-03-29T17:49:53.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,857 [shard 0] monc - build_monmap: for_mkfs=false 2022-03-29T17:49:53.860 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,857 [shard 0] monc - reopen_session to mon.-1 2022-03-29T17:49:53.860 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:53,857 [shard 0] monc - mon.0 does not have an addr compatible with me 2022-03-29T17:49:53.860 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,857 [shard 0] monc - connecting to mon.1 2022-03-29T17:49:53.860 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,857 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6805/35577 >> mon.? v2:172.21.15.138:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:3300/0, peer_name=mon.?, cc=10528021445568868744 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:53.860 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,857 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6805/35577 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:49:53.861 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,857 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6805/35577@51218 >> mon.? v2:172.21.15.138:3300/0] peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:51218/0 (socket says 172.21.15.138:51218) 2022-03-29T17:49:53.861 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,857 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@51218 >> mon.? v2:172.21.15.138:3300/0] learned myaddr=v2:172.21.15.138:6805/35577 (blank IP) 2022-03-29T17:49:53.861 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,858 [shard 0] monc - get_auth_request(con=[osd.0(client) v2:172.21.15.138:6805/35577@51218 >> mon.? v2:172.21.15.138:3300/0], auth_method=0) 2022-03-29T17:49:53.861 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,858 [shard 0] monc - method 2 preferred_modes {1} 2022-03-29T17:49:53.865 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,864 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@58384 >> mon.0 v2:172.21.15.138:3300/0] connected: gs=1, pgs=25, cs=0, client_cookie=8058926782284612243, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2022-03-29T17:49:53.867 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,864 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2022-03-29T17:49:53.867 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,864 [shard 0] monc - handle_monmap: renewing tickets 2022-03-29T17:49:53.867 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,864 [shard 0] monc - renew_rotating_keyring called too often 2022-03-29T17:49:53.867 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,865 [shard 0] monc - handle_mon_map: renewed tickets 2022-03-29T17:49:53.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,865 [shard 0] monc - handle_auth_reply mon v2:172.21.15.138:6800/35578 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,865 [shard 0] monc - handle_auth_reply 2022-03-29T17:49:53.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,865 [shard 0] monc - do_auth_single: mon v2:172.21.15.138:6800/35578 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,865 [shard 0] alienstore - stat 2022-03-29T17:49:53.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:53,865 [shard 0] osd - operator() crush location is "host=smithi138", "root=default" 2022-03-29T17:49:53.890 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,888 [shard 0] monc - responding with 36 bytes 2022-03-29T17:49:53.891 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,889 [shard 0] monc - found mon.noname-a 2022-03-29T17:49:53.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:53,889 [shard 0] monc - reopen_session mon connection attempts complete 2022-03-29T17:49:53.891 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,889 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2022-03-29T17:49:53.891 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,889 [shard 0] monc - waiting 2022-03-29T17:49:53.899 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,898 [shard 0] monc - responding with 36 bytes 2022-03-29T17:49:53.900 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,899 [shard 0] monc - found mon.noname-a 2022-03-29T17:49:53.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:53,899 [shard 0] monc - reopen_session mon connection attempts complete 2022-03-29T17:49:53.901 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,899 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2022-03-29T17:49:53.901 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,899 [shard 0] monc - waiting 2022-03-29T17:49:53.910 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:53.907+0000 7fb027daaf40 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-03-29T17:49:53.931 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,930 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@56603 >> mon.0 v2:172.21.15.138:3300/0] connected: gs=1, pgs=26, cs=0, client_cookie=12217060769435700643, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2022-03-29T17:49:53.933 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2022-03-29T17:49:53.933 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] monc - handle_monmap: renewing tickets 2022-03-29T17:49:53.933 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] monc - renew_rotating_keyring called too often 2022-03-29T17:49:53.934 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] monc - handle_mon_map: renewed tickets 2022-03-29T17:49:53.934 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] monc - handle_auth_reply mon v2:172.21.15.138:6803/35579 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.934 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] monc - handle_auth_reply 2022-03-29T17:49:53.934 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] monc - do_auth_single: mon v2:172.21.15.138:6803/35579 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.935 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] alienstore - stat 2022-03-29T17:49:53.935 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:53,931 [shard 0] osd - operator() crush location is "host=smithi138", "root=default" 2022-03-29T17:49:53.940 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,939 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@51218 >> mon.0 v2:172.21.15.138:3300/0] connected: gs=1, pgs=27, cs=0, client_cookie=10528021445568868744, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2022-03-29T17:49:53.942 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,939 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2022-03-29T17:49:53.942 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,939 [shard 0] monc - handle_monmap: renewing tickets 2022-03-29T17:49:53.942 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,939 [shard 0] monc - renew_rotating_keyring called too often 2022-03-29T17:49:53.943 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,939 [shard 0] monc - handle_mon_map: renewed tickets 2022-03-29T17:49:53.943 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,940 [shard 0] monc - handle_auth_reply mon v2:172.21.15.138:6805/35577 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.943 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,940 [shard 0] monc - handle_auth_reply 2022-03-29T17:49:53.943 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,940 [shard 0] monc - do_auth_single: mon v2:172.21.15.138:6805/35577 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:49:53.944 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,940 [shard 0] alienstore - stat 2022-03-29T17:49:53.944 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:53,940 [shard 0] osd - operator() crush location is "host=smithi138", "root=default" 2022-03-29T17:49:54.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:54.071+0000 7fb027daaf40 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-03-29T17:49:54.183 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:54.180+0000 7fb027daaf40 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-03-29T17:49:54.210 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,207 [shard 0] osd - added to crush: create-or-move updating item name 'osd.1' weight 0.0977 at location {host=smithi138,root=default} to crush map 2022-03-29T17:49:54.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:54,208 [shard 0] osd - replacing unkwnown addrs of cluster messenger 2022-03-29T17:49:54.210 INFO:tasks.ceph.osd.1.smithi138.stderr:ERROR 2022-03-29 17:49:54,208 [shard 0] none - Falling back to public interface 2022-03-29T17:49:54.210 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,208 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:54.211 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,208 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:54.211 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,208 [shard 0] osd - heartbeat: start front_addrs=v2:0.0.0.0:0/0, back_addrs=v2:0.0.0.0:0/0 2022-03-29T17:49:54.212 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,208 [shard 0] osd - added to crush: create-or-move updating item name 'osd.0' weight 0.0977 at location {host=smithi138,root=default} to crush map 2022-03-29T17:49:54.212 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:54,209 [shard 0] osd - replacing unkwnown addrs of cluster messenger 2022-03-29T17:49:54.213 INFO:tasks.ceph.osd.0.smithi138.stderr:ERROR 2022-03-29 17:49:54,209 [shard 0] none - Falling back to public interface 2022-03-29T17:49:54.213 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,209 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:54.213 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,209 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:54.213 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,209 [shard 0] osd - heartbeat: start front_addrs=v2:0.0.0.0:0/0, back_addrs=v2:0.0.0.0:0/0 2022-03-29T17:49:54.214 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:54,211 [shard 0] mgrc - No active mgr available yet 2022-03-29T17:49:54.214 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:54,212 [shard 0] mgrc - No active mgr available yet 2022-03-29T17:49:54.215 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,212 [shard 0] ms - [osd.0(hb_back) v2:0.0.0.0:6806/35577] try_bind: done 2022-03-29T17:49:54.215 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,213 [shard 0] ms - [osd.0(hb_back) v2:0.0.0.0:6806/35577] try_bind: done 2022-03-29T17:49:54.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,213 [shard 0] osd - handle_osd_map osd_map(6..6 src has 1..6) v4 2022-03-29T17:49:54.216 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:54,213 [shard 0] osd - i am still initializing 2022-03-29T17:49:54.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,213 [shard 0] ms - [osd.1(hb_back) v2:0.0.0.0:6807/35578] try_bind: done 2022-03-29T17:49:54.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,213 [shard 0] ms - [osd.1(hb_back) v2:0.0.0.0:6807/35578] try_bind: done 2022-03-29T17:49:54.217 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,213 [shard 0] osd - handle_osd_map osd_map(6..6 src has 1..6) v4 2022-03-29T17:49:54.217 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:54,213 [shard 0] osd - i am still initializing 2022-03-29T17:49:54.220 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] ms - [osd.1(hb_front) v2:0.0.0.0:6808/35578] try_bind: done 2022-03-29T17:49:54.220 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] ms - [osd.1(hb_front) v2:0.0.0.0:6808/35578] try_bind: done 2022-03-29T17:49:54.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:54,214 [shard 0] osd - start: asok socket path=/var/run/ceph/ceph-osd.1.asok 2022-03-29T17:49:54.220 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): version) 2022-03-29T17:49:54.221 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): git_version) 2022-03-29T17:49:54.221 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): help) 2022-03-29T17:49:54.221 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): get_command_descriptions) 2022-03-29T17:49:54.221 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): config get) 2022-03-29T17:49:54.221 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): config set) 2022-03-29T17:49:54.222 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): config show) 2022-03-29T17:49:54.222 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): config help) 2022-03-29T17:49:54.222 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): injectargs) 2022-03-29T17:49:54.222 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): status) 2022-03-29T17:49:54.223 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): send_beacon) 2022-03-29T17:49:54.223 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): flush_pg_stats) 2022-03-29T17:49:54.223 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): dump_pgstate_history) 2022-03-29T17:49:54.223 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): dump_metrics) 2022-03-29T17:49:54.224 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): perfcounters_dump) 2022-03-29T17:49:54.224 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): injectdataerr) 2022-03-29T17:49:54.224 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): injectmdataerr) 2022-03-29T17:49:54.224 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): query) 2022-03-29T17:49:54.224 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): mark_unfound_lost) 2022-03-29T17:49:54.227 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] ms - [osd.0(hb_front) v2:0.0.0.0:6809/35577] try_bind: done 2022-03-29T17:49:54.227 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] ms - [osd.0(hb_front) v2:0.0.0.0:6809/35577] try_bind: done 2022-03-29T17:49:54.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:54,214 [shard 0] osd - start: asok socket path=/var/run/ceph/ceph-osd.0.asok 2022-03-29T17:49:54.227 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,214 [shard 0] osd - register_command(): version) 2022-03-29T17:49:54.227 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): git_version) 2022-03-29T17:49:54.228 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): help) 2022-03-29T17:49:54.228 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): get_command_descriptions) 2022-03-29T17:49:54.228 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): config get) 2022-03-29T17:49:54.228 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): config set) 2022-03-29T17:49:54.229 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): config show) 2022-03-29T17:49:54.229 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): config help) 2022-03-29T17:49:54.229 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): injectargs) 2022-03-29T17:49:54.229 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): status) 2022-03-29T17:49:54.229 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): send_beacon) 2022-03-29T17:49:54.230 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): flush_pg_stats) 2022-03-29T17:49:54.230 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): dump_pgstate_history) 2022-03-29T17:49:54.230 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): dump_metrics) 2022-03-29T17:49:54.230 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): perfcounters_dump) 2022-03-29T17:49:54.230 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): injectdataerr) 2022-03-29T17:49:54.231 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): injectmdataerr) 2022-03-29T17:49:54.231 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): query) 2022-03-29T17:49:54.231 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - register_command(): mark_unfound_lost) 2022-03-29T17:49:54.232 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - osd.1: _preboot 2022-03-29T17:49:54.232 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - waiting for initial osdmap 2022-03-29T17:49:54.232 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - osdmap_subscribe(1) 2022-03-29T17:49:54.232 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - crimson startup completed 2022-03-29T17:49:54.233 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - osd.0: _preboot 2022-03-29T17:49:54.233 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - waiting for initial osdmap 2022-03-29T17:49:54.234 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,215 [shard 0] osd - osdmap_subscribe(1) 2022-03-29T17:49:54.234 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,216 [shard 0] osd - crimson startup completed 2022-03-29T17:49:54.234 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,216 [shard 0] osd - handle_osd_map osd_map(1..6 src has 1..6) v4 2022-03-29T17:49:54.235 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,216 [shard 0] osd - handle_osd_map epochs [1..6], i have 0, src has [1..6] 2022-03-29T17:49:54.235 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,216 [shard 0] osd - handle_osd_map osd_map(1..6 src has 1..6) v4 2022-03-29T17:49:54.235 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,216 [shard 0] osd - handle_osd_map epochs [1..6], i have 0, src has [1..6] 2022-03-29T17:49:54.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,218 [shard 0] osd - added to crush: create-or-move updating item name 'osd.2' weight 0.0977 at location {host=smithi138,root=default} to crush map 2022-03-29T17:49:54.236 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:54,218 [shard 0] osd - replacing unkwnown addrs of cluster messenger 2022-03-29T17:49:54.237 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:49:54,219 [shard 0] none - Falling back to public interface 2022-03-29T17:49:54.237 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,219 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:54.237 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,219 [shard 0] osd - picked address v2:0.0.0.0:0/0 2022-03-29T17:49:54.237 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,219 [shard 0] osd - heartbeat: start front_addrs=v2:0.0.0.0:0/0, back_addrs=v2:0.0.0.0:0/0 2022-03-29T17:49:54.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:54,220 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:54.238 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:54,220 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:54.239 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:54,222 [shard 0] mgrc - No active mgr available yet 2022-03-29T17:49:54.239 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,224 [shard 0] osd - handle_osd_map osd_map(6..6 src has 1..6) v4 2022-03-29T17:49:54.239 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:54,224 [shard 0] osd - i am still initializing 2022-03-29T17:49:54.240 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,224 [shard 0] osd - osd.1: committed_osd_maps(1, 6) 2022-03-29T17:49:54.240 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,224 [shard 0] osd - osd.0: committed_osd_maps(1, 6) 2022-03-29T17:49:54.241 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - osd.1: now preboot 2022-03-29T17:49:54.241 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - osd.1: _preboot 2022-03-29T17:49:54.241 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - hb_back_msgr: v2:172.21.15.138:6807/35578 2022-03-29T17:49:54.242 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - hb_front_msgr: v2:172.21.15.138:6808/35578 2022-03-29T17:49:54.242 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - cluster_msgr: v2:172.21.15.138:6801/35578 2022-03-29T17:49:54.243 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - osd.0: now preboot 2022-03-29T17:49:54.243 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - osd.0: _preboot 2022-03-29T17:49:54.243 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - hb_back_msgr: v2:172.21.15.138:6806/35577 2022-03-29T17:49:54.243 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - hb_front_msgr: v2:172.21.15.138:6809/35577 2022-03-29T17:49:54.244 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:54,225 [shard 0] osd - cluster_msgr: v2:172.21.15.138:6804/35577 2022-03-29T17:49:54.247 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] ms - [osd.2(hb_front) v2:0.0.0.0:6810/35579] try_bind: done 2022-03-29T17:49:54.247 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] ms - [osd.2(hb_front) v2:0.0.0.0:6810/35579] try_bind: done 2022-03-29T17:49:54.247 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] ms - [osd.2(hb_back) v2:0.0.0.0:6811/35579] try_bind: done 2022-03-29T17:49:54.248 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] ms - [osd.2(hb_back) v2:0.0.0.0:6811/35579] try_bind: done 2022-03-29T17:49:54.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:54,227 [shard 0] osd - start: asok socket path=/var/run/ceph/ceph-osd.2.asok 2022-03-29T17:49:54.248 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): version) 2022-03-29T17:49:54.248 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): git_version) 2022-03-29T17:49:54.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): help) 2022-03-29T17:49:54.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): get_command_descriptions) 2022-03-29T17:49:54.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): config get) 2022-03-29T17:49:54.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): config set) 2022-03-29T17:49:54.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): config show) 2022-03-29T17:49:54.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): config help) 2022-03-29T17:49:54.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): injectargs) 2022-03-29T17:49:54.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): status) 2022-03-29T17:49:54.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): send_beacon) 2022-03-29T17:49:54.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): flush_pg_stats) 2022-03-29T17:49:54.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): dump_pgstate_history) 2022-03-29T17:49:54.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): dump_metrics) 2022-03-29T17:49:54.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): perfcounters_dump) 2022-03-29T17:49:54.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): injectdataerr) 2022-03-29T17:49:54.252 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): injectmdataerr) 2022-03-29T17:49:54.252 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): query) 2022-03-29T17:49:54.252 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - register_command(): mark_unfound_lost) 2022-03-29T17:49:54.252 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,227 [shard 0] osd - osd.2: _preboot 2022-03-29T17:49:54.253 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,228 [shard 0] osd - waiting for initial osdmap 2022-03-29T17:49:54.253 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,228 [shard 0] osd - osdmap_subscribe(1) 2022-03-29T17:49:54.253 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,228 [shard 0] osd - crimson startup completed 2022-03-29T17:49:54.253 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,228 [shard 0] osd - handle_osd_map osd_map(1..6 src has 1..6) v4 2022-03-29T17:49:54.254 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,228 [shard 0] osd - handle_osd_map epochs [1..6], i have 0, src has [1..6] 2022-03-29T17:49:54.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:54,230 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:54.254 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,233 [shard 0] osd - osd.2: committed_osd_maps(1, 6) 2022-03-29T17:49:54.254 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,234 [shard 0] osd - osd.2: now preboot 2022-03-29T17:49:54.254 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,234 [shard 0] osd - osd.2: _preboot 2022-03-29T17:49:54.255 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,234 [shard 0] osd - hb_back_msgr: v2:172.21.15.138:6811/35579 2022-03-29T17:49:54.255 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,234 [shard 0] osd - hb_front_msgr: v2:172.21.15.138:6810/35579 2022-03-29T17:49:54.255 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:54,234 [shard 0] osd - cluster_msgr: v2:172.21.15.138:6802/35579 2022-03-29T17:49:54.288 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:54.285+0000 7fb027daaf40 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-03-29T17:49:54.969 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:54.966+0000 7fb027daaf40 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-03-29T17:49:55.210 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:55,208 [shard 0] osd - handle_osd_map osd_map(7..7 src has 1..7) v4 2022-03-29T17:49:55.210 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:55,208 [shard 0] osd - handle_osd_map epochs [7..7], i have 6, src has [1..7] 2022-03-29T17:49:55.211 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:55,209 [shard 0] osd - handle_osd_map osd_map(7..7 src has 1..7) v4 2022-03-29T17:49:55.211 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:55,209 [shard 0] osd - handle_osd_map epochs [7..7], i have 6, src has [1..7] 2022-03-29T17:49:55.212 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:55,209 [shard 0] osd - handle_osd_map osd_map(6..7 src has 1..7) v4 2022-03-29T17:49:55.212 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:55,209 [shard 0] osd - handle_osd_map epochs [6..7], i have 6, src has [1..7] 2022-03-29T17:49:55.213 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:55,210 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:55.213 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:55,211 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:55.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:55,211 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:55.214 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:55,212 [shard 0] osd - osd.2: committed_osd_maps(7, 7) 2022-03-29T17:49:55.215 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:55,212 [shard 0] osd - osd.2: map e 7 marked me up: up_from 7, bind_epoch 0, state booting 2022-03-29T17:49:55.215 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - osd.2: activating... 2022-03-29T17:49:55.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:55,212 [shard 0] osd - handle_osd_map osd_map(7..7 src has 1..7) v4 2022-03-29T17:49:55.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - handle_osd_map epochs [7..7], i have 7, src has [1..7] 2022-03-29T17:49:55.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - osd.1: committed_osd_maps(7, 7) 2022-03-29T17:49:55.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - osd.1: map e 7 marked me up: up_from 7, bind_epoch 0, state booting 2022-03-29T17:49:55.217 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - osd.1: activating... 2022-03-29T17:49:55.217 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - osd.0: committed_osd_maps(7, 7) 2022-03-29T17:49:55.218 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - osd.0: map e 7 marked me up: up_from 7, bind_epoch 0, state booting 2022-03-29T17:49:55.218 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - osd.0: activating... 2022-03-29T17:49:55.218 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:55,213 [shard 0] osd - osd.2: now active 2022-03-29T17:49:55.219 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:55,214 [shard 0] osd - osd.1: now active 2022-03-29T17:49:55.219 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:55,214 [shard 0] osd - osd.0: now active 2022-03-29T17:49:55.221 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:55.218+0000 7fb027daaf40 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-03-29T17:49:55.785 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:55.783+0000 7fb027daaf40 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-03-29T17:49:55.914 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:55.912+0000 7fb027daaf40 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-03-29T17:49:56.211 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,208 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=16486000033257166439 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.211 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,209 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=18063609415521801884 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.211 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,209 [shard 0] osd - Heartbeat::Peer: osd.0 added 2022-03-29T17:49:56.212 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,209 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.2 v2:172.21.15.138:6810/35579] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6810/35579, peer_name=osd.2, cc=13001536588210862930 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.212 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,209 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.2 v2:172.21.15.138:6811/35579] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6811/35579, peer_name=osd.2, cc=3272032577814993038 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.212 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,209 [shard 0] osd - Heartbeat::Peer: osd.2 added 2022-03-29T17:49:56.212 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,209 [shard 0] alienstore - stat 2022-03-29T17:49:56.213 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,209 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> unknown.? -@51453] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:51453/0 2022-03-29T17:49:56.213 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,210 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> unknown.? -@59901] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:59901/0 2022-03-29T17:49:56.213 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,210 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> unknown.? -@53604] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:53604/0 2022-03-29T17:49:56.214 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,210 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> unknown.? -@58329] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:58329/0 2022-03-29T17:49:56.215 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,213 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.? -@53604] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.215 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,214 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.? -@51453] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.216 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,214 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.? -@59901] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.217 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=4578994475146889057 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.217 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=4171044242327705109 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.217 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] osd - Heartbeat::Peer: osd.1 added 2022-03-29T17:49:56.218 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.2 v2:172.21.15.138:6810/35579] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6810/35579, peer_name=osd.2, cc=843340835583433663 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.218 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.2 v2:172.21.15.138:6811/35579] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6811/35579, peer_name=osd.2, cc=11388434643464638206 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.218 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] osd - Heartbeat::Peer: osd.2 added 2022-03-29T17:49:56.218 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] alienstore - stat 2022-03-29T17:49:56.219 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=18112009746374483298 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.220 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=5641693999390928170 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.220 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] osd - Heartbeat::Peer: osd.0 added 2022-03-29T17:49:56.220 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,215 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=16976046876214090229 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.220 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,216 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=5285492355575901828 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:56.221 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,216 [shard 0] osd - Heartbeat::Peer: osd.1 added 2022-03-29T17:49:56.221 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,216 [shard 0] alienstore - stat 2022-03-29T17:49:56.222 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,216 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> unknown.? -@55215] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:55215/0 2022-03-29T17:49:56.222 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,216 [shard 0] monc - get_auth_request(con=[osd.1(hb_back) v2:172.21.15.138:6807/35578@51453 >> osd.2 v2:172.21.15.138:6811/35579], auth_method=0) 2022-03-29T17:49:56.222 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,216 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> unknown.? -@64528] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:64528/0 2022-03-29T17:49:56.223 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,217 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> unknown.? -@50045] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:50045/0 2022-03-29T17:49:56.223 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,217 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> unknown.? -@52708] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:52708/0 2022-03-29T17:49:56.223 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,217 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> unknown.? -@61441] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:61441/0 2022-03-29T17:49:56.224 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,218 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> unknown.? -@54689] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:54689/0 2022-03-29T17:49:56.224 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,218 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> unknown.? -@53746] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:53746/0 2022-03-29T17:49:56.225 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,218 [shard 0] monc - get_auth_request(con=[osd.1(hb_back) v2:172.21.15.138:6807/35578@53604 >> osd.0 v2:172.21.15.138:6806/35577], auth_method=0) 2022-03-29T17:49:56.225 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,218 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> unknown.? -@58199] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:58199/0 2022-03-29T17:49:56.225 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,219 [shard 0] monc - get_auth_request(con=[osd.1(hb_front) v2:172.21.15.138:6808/35578@59901 >> osd.0 v2:172.21.15.138:6809/35577], auth_method=0) 2022-03-29T17:49:56.226 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,220 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.? -@58329] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.227 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,220 [shard 0] monc - get_auth_request(con=[osd.1(hb_front) v2:172.21.15.138:6808/35578@58329 >> osd.2 v2:172.21.15.138:6810/35579], auth_method=0) 2022-03-29T17:49:56.227 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,222 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.? -@55215] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.227 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,222 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.? -@53746] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.228 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,222 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.? -@52708] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.228 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,223 [shard 0] monc - get_auth_request(con=[osd.2(hb_back) v2:172.21.15.138:6811/35579@50045 >> osd.0 v2:172.21.15.138:6806/35577], auth_method=0) 2022-03-29T17:49:56.229 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,223 [shard 0] monc - get_auth_request(con=[osd.0(hb_front) v2:172.21.15.138:6809/35577@52708 >> osd.2 v2:172.21.15.138:6810/35579], auth_method=0) 2022-03-29T17:49:56.229 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,223 [shard 0] monc - get_auth_request(con=[osd.0(hb_back) v2:172.21.15.138:6806/35577@55215 >> osd.1 v2:172.21.15.138:6807/35578], auth_method=0) 2022-03-29T17:49:56.229 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,223 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.? -@54689] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.230 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,224 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.? -@64528] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.230 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,224 [shard 0] monc - added challenge on [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.? -@55215] 2022-03-29T17:49:56.231 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,224 [shard 0] monc - get_auth_request(con=[osd.2(hb_front) v2:172.21.15.138:6810/35579@64528 >> osd.1 v2:172.21.15.138:6808/35578], auth_method=0) 2022-03-29T17:49:56.231 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,224 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.? -@61441] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.231 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,224 [shard 0] monc - get_auth_request(con=[osd.2(hb_back) v2:172.21.15.138:6811/35579@53746 >> osd.1 v2:172.21.15.138:6807/35578], auth_method=0) 2022-03-29T17:49:56.232 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,224 [shard 0] monc - get_auth_request(con=[osd.0(hb_back) v2:172.21.15.138:6806/35577@61441 >> osd.2 v2:172.21.15.138:6811/35579], auth_method=0) 2022-03-29T17:49:56.232 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,225 [shard 0] monc - get_auth_request(con=[osd.0(hb_front) v2:172.21.15.138:6809/35577@54689 >> osd.1 v2:172.21.15.138:6808/35578], auth_method=0) 2022-03-29T17:49:56.232 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,225 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.? -@58199] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.233 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,225 [shard 0] monc - added challenge on [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.? -@53746] 2022-03-29T17:49:56.233 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,225 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.? -@50045] UPDATE: peer_type=osd, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:49:56.234 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,226 [shard 0] monc - get_auth_request(con=[osd.2(hb_front) v2:172.21.15.138:6810/35579@58199 >> osd.0 v2:172.21.15.138:6809/35577], auth_method=0) 2022-03-29T17:49:56.234 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,226 [shard 0] monc - added challenge on [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.? -@52708] 2022-03-29T17:49:56.234 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,227 [shard 0] monc - added challenge on [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.? -@61441] 2022-03-29T17:49:56.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:56,227 [shard 0] osd - handle_authentication osd.0 has caps allow * 2022-03-29T17:49:56.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:56,228 [shard 0] osd - handle_authentication osd.2 has caps allow * 2022-03-29T17:49:56.236 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:56,229 [shard 0] osd - handle_authentication osd.0 has caps allow * 2022-03-29T17:49:56.236 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:56,230 [shard 0] osd - handle_authentication osd.0 has caps allow * 2022-03-29T17:49:56.237 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:56,231 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577@55215] server_connect: existing connection [osd.1(hb_back) v2:172.21.15.138:6807/35578@53604 >> osd.0 v2:172.21.15.138:6806/35577] is a lossy channel. Close existing in favor of this connection 2022-03-29T17:49:56.237 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,231 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578@53604 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset yes, replace yes 2022-03-29T17:49:56.237 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:56,231 [shard 0] osd - Heartbeat::Connection::replaced(): con_back(osd.0) racing 2022-03-29T17:49:56.238 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,231 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577@55215] established: gs=0, pgs=1, cs=0, client_cookie=4171044242327705109, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.238 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,232 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577@55215 >> osd.1 v2:172.21.15.138:6807/35578] connected: gs=1, pgs=3, cs=0, client_cookie=4171044242327705109, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.238 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,232 [shard 0] monc - added challenge on [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.? -@53604] 2022-03-29T17:49:56.239 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:56,232 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577@52708] server_connect: existing connection [osd.2(hb_front) v2:172.21.15.138:6810/35579@58199 >> osd.0 v2:172.21.15.138:6809/35577] is a lossy channel. Close existing in favor of this connection 2022-03-29T17:49:56.239 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,232 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579@58199 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset yes, replace yes 2022-03-29T17:49:56.240 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:56,232 [shard 0] osd - Heartbeat::Connection::replaced(): con_front(osd.0) racing 2022-03-29T17:49:56.241 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:56,232 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.2 v2:172.21.15.138:6811/35579@53746] server_connect: existing connection [osd.1(hb_back) v2:172.21.15.138:6807/35578@51453 >> osd.2 v2:172.21.15.138:6811/35579] is a lossy channel. Close existing in favor of this connection 2022-03-29T17:49:56.241 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,233 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578@51453 >> osd.2 v2:172.21.15.138:6811/35579] closing: reset yes, replace yes 2022-03-29T17:49:56.241 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:56,233 [shard 0] osd - Heartbeat::Connection::replaced(): con_back(osd.2) racing 2022-03-29T17:49:56.241 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,233 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578@53604 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.242 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,233 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577@52708] established: gs=0, pgs=2, cs=0, client_cookie=843340835583433663, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.242 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,234 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579@58199 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.243 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,234 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.2 v2:172.21.15.138:6811/35579@53746] established: gs=0, pgs=2, cs=0, client_cookie=5285492355575901828, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.243 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,234 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578@51453 >> osd.2 v2:172.21.15.138:6811/35579] execute_connecting(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.244 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,233 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.? -@53604] execute_accepting(): fault at ACCEPTING, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.244 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,234 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.? -@53604] closing: reset no, replace no 2022-03-29T17:49:56.244 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,234 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577@52708 >> osd.2 v2:172.21.15.138:6810/35579] connected: gs=2, pgs=3, cs=0, client_cookie=843340835583433663, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.245 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,235 [shard 0] monc - added challenge on [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.? -@58199] 2022-03-29T17:49:56.245 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,235 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579@53746 >> osd.1 v2:172.21.15.138:6807/35578] connected: gs=2, pgs=4, cs=0, client_cookie=5285492355575901828, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.246 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,235 [shard 0] monc - added challenge on [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.? -@51453] 2022-03-29T17:49:56.246 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,235 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.? -@51453] execute_accepting(): fault at ACCEPTING, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.246 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,235 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.? -@51453] closing: reset no, replace no 2022-03-29T17:49:56.247 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,236 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.? -@58199] execute_accepting(): fault at ACCEPTING, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.247 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,236 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.? -@58199] closing: reset no, replace no 2022-03-29T17:49:56.248 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:56,236 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577@61441] server_connect: existing connection [osd.2(hb_back) v2:172.21.15.138:6811/35579@50045 >> osd.0 v2:172.21.15.138:6806/35577] is a lossy channel. Close existing in favor of this connection 2022-03-29T17:49:56.248 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,236 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579@50045 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset yes, replace yes 2022-03-29T17:49:56.248 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:56,236 [shard 0] osd - Heartbeat::Connection::replaced(): con_back(osd.0) racing 2022-03-29T17:49:56.248 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,236 [shard 0] osd - Heartbeat::Peer: osd.0 connected (send=false) 2022-03-29T17:49:56.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,236 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577@61441] established: gs=0, pgs=2, cs=0, client_cookie=11388434643464638206, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.249 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,236 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577@61441 >> osd.2 v2:172.21.15.138:6811/35579] connected: gs=2, pgs=3, cs=0, client_cookie=11388434643464638206, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.250 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,237 [shard 0] osd - Heartbeat::Peer: osd.2 connected (send=false) 2022-03-29T17:49:56.250 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,237 [shard 0] monc - added challenge on [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.? -@50045] 2022-03-29T17:49:56.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,237 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579@50045 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.251 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,237 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.? -@50045] execute_accepting(): fault at ACCEPTING, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.251 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,237 [shard 0] ms - [osd.0(hb_back) v2:172.21.15.138:6806/35577 >> osd.? -@50045] closing: reset no, replace no 2022-03-29T17:49:56.256 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,255 [shard 0] monc - added challenge on [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.? -@58329] 2022-03-29T17:49:56.257 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,256 [shard 0] monc - added challenge on [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.? -@59901] 2022-03-29T17:49:56.258 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:49:56,257 [shard 0] osd - handle_authentication osd.1 has caps allow * 2022-03-29T17:49:56.259 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:56,258 [shard 0] osd - handle_authentication osd.1 has caps allow * 2022-03-29T17:49:56.261 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:56,260 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.1 v2:172.21.15.138:6808/35578@59901] server_connect: existing connection [osd.0(hb_front) v2:172.21.15.138:6809/35577@54689 >> osd.1 v2:172.21.15.138:6808/35578] is a lossy channel. Close existing in favor of this connection 2022-03-29T17:49:56.262 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,260 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577@54689 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset yes, replace yes 2022-03-29T17:49:56.262 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:56,260 [shard 0] osd - Heartbeat::Connection::replaced(): con_front(osd.1) racing 2022-03-29T17:49:56.262 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,260 [shard 0] osd - Heartbeat::Peer: osd.1 connected (send=false) 2022-03-29T17:49:56.263 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,260 [shard 0] monc - added challenge on [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.? -@54689] 2022-03-29T17:49:56.263 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,260 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.? -@54689] execute_accepting(): fault at ACCEPTING, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.263 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,260 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.? -@54689] closing: reset no, replace no 2022-03-29T17:49:56.264 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:56,261 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578@58329] server_connect: existing connection [osd.2(hb_front) v2:172.21.15.138:6810/35579@64528 >> osd.1 v2:172.21.15.138:6808/35578] is a lossy channel. Close existing in favor of this connection 2022-03-29T17:49:56.264 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,261 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579@64528 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset yes, replace yes 2022-03-29T17:49:56.264 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:56,261 [shard 0] osd - Heartbeat::Connection::replaced(): con_front(osd.1) racing 2022-03-29T17:49:56.265 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,261 [shard 0] osd - Heartbeat::Peer: osd.1 connected (send=false) 2022-03-29T17:49:56.265 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,261 [shard 0] monc - added challenge on [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.? -@64528] 2022-03-29T17:49:56.266 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,261 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578@58329 >> osd.2 v2:172.21.15.138:6810/35579] connected: gs=2, pgs=4, cs=0, client_cookie=13001536588210862930, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.266 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,261 [shard 0] osd - Heartbeat::Peer: osd.2 connected (send=false) 2022-03-29T17:49:56.267 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,261 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577 >> osd.1 v2:172.21.15.138:6808/35578@59901] established: gs=0, pgs=1, cs=0, client_cookie=16486000033257166439, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.267 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:56,261 [shard 0] ms - [osd.0(hb_front) v2:172.21.15.138:6809/35577@54689 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.267 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,261 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579@64528 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.268 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:56,262 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578@58329] established: gs=0, pgs=2, cs=0, client_cookie=13001536588210862930, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.268 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,262 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578@59901 >> osd.0 v2:172.21.15.138:6809/35577] connected: gs=1, pgs=3, cs=0, client_cookie=16486000033257166439, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:56.269 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,262 [shard 0] osd - Heartbeat::Peer: osd.0 connected (send=false) 2022-03-29T17:49:56.269 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,263 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.? -@64528] execute_accepting(): fault at ACCEPTING, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:56.269 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:56,263 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.? -@64528] closing: reset no, replace no 2022-03-29T17:49:56.269 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:56.265+0000 7fb027daaf40 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-03-29T17:49:56.343 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:56.340+0000 7fb027daaf40 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-03-29T17:49:56.414 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:56.411+0000 7fb027daaf40 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-03-29T17:49:56.495 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:56.493+0000 7fb027daaf40 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-03-29T17:49:56.570 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:56.567+0000 7fb027daaf40 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-03-29T17:49:56.668 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:49:56.665+0000 7fb027daaf40 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-03-29T17:49:56.694 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:49:56,692 [shard 0] mgrc - No active mgr available yet 2022-03-29T17:49:56.695 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:56,693 [shard 0] mgrc - No active mgr available yet 2022-03-29T17:49:56.695 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:49:56,694 [shard 0] mgrc - No active mgr available yet 2022-03-29T17:49:57.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,215 [shard 0] alienstore - stat 2022-03-29T17:49:57.217 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,215 [shard 0] alienstore - stat 2022-03-29T17:49:57.218 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:57,216 [shard 0] alienstore - stat 2022-03-29T17:49:57.699 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,697 [shard 0] osd - handle_osd_map osd_map(7..8 src has 1..8) v4 2022-03-29T17:49:57.699 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,697 [shard 0] osd - handle_osd_map epochs [7..8], i have 7, src has [1..8] 2022-03-29T17:49:57.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,698 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:57.701 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,698 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e8 {1.0=8,2022-03-29T17:49:57.683718+0000}) v2): starting 2022-03-29T17:49:57.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,698 [shard 0] osd - handle_pg_create: 1.0 e8 @1648576197.683719 history ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8 pi ([0,0] all_participants= intervals=) 2022-03-29T17:49:57.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,699 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 NullEvt +create_info)): start 2022-03-29T17:49:57.702 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,699 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e8 {1.0=8,2022-03-29T17:49:57.683718+0000}) v2): about to fork future 2022-03-29T17:49:57.702 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,699 [shard 0] osd - evt epoch is 8, i have 7, will wait 2022-03-29T17:49:57.702 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,699 [shard 0] osd - osdmap_subscribe(7) 2022-03-29T17:49:57.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,700 [shard 0] osd - handle_osd_map osd_map(8..8 src has 1..8) v4 2022-03-29T17:49:57.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,700 [shard 0] osd - handle_osd_map epochs [8..8], i have 8, src has [1..8] 2022-03-29T17:49:57.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,700 [shard 0] osd - osd.1: committed_osd_maps(8, 8) 2022-03-29T17:49:57.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,700 [shard 0] osd - osd.1: map e 8 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:49:57.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,700 [shard 0] osd - osd.1: now active 2022-03-29T17:49:57.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,700 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 NullEvt +create_info)): got map 8 2022-03-29T17:49:57.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,700 [shard 0] osd - Creating 1.0 2022-03-29T17:49:57.714 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,701 [shard 0] osd - pg_epoch 8 pg[1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:49:57.714 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,701 [shard 0] osd - Entering state: Initial 2022-03-29T17:49:57.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,701 [shard 0] osd - pg_epoch 8 pg[1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:49:57.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,701 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:49:57.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,701 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:49:57.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,701 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:49:57.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,701 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:49:57.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,701 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 4.397737503s + 0.000000000s) 2022-03-29T17:49:57.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:57.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_advance_map(id=1, detail=PGAdvanceMap(pg=1.0 from=8 to=8 do_init)): start 2022-03-29T17:49:57.717 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000781 0 0.000000 2022-03-29T17:49:57.717 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Exiting state: Initial, entered at 1648576197.7013564, 0.0 spent on 0 events 2022-03-29T17:49:57.717 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:49:57.718 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Entering state: Reset 2022-03-29T17:49:57.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 8 2022-03-29T17:49:57.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:49:57.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:49:57.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:49:57.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:49:57.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown exit Reset 0.000224 1 2022-03-29T17:49:57.702391+0000 2022-03-29T17:49:57.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Exiting state: Reset, entered at 1648576197.7022057, 1648576197.7023914 spent on 1 events 2022-03-29T17:49:57.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:49:57.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Entering state: Started 2022-03-29T17:49:57.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:49:57.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Entering state: Start 2022-03-29T17:49:57.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:49:57.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown exit Start 0.000068 0 0.000000 2022-03-29T17:49:57.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Exiting state: Start, entered at 1648576197.702521, 0.0 spent on 0 events 2022-03-29T17:49:57.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:49:57.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:49:57.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:49:57.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:49:57.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:49:57.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,702 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:49:57.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:49:57.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:49:57.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering up_thru 0 < same_since 8, must notify monitor 2022-03-29T17:49:57.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:49:57.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:49:57.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,702 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:57.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:49:57.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 8 while current is 8 2022-03-29T17:49:57.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - log is not dirty 2022-03-29T17:49:57.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:49:57.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:49:57.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:57.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 8:1 2022-03-29T17:49:57.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 8 while current is 8 2022-03-29T17:49:57.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - log is not dirty 2022-03-29T17:49:57.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - Created 1.0 2022-03-29T17:49:57.727 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,703 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:49:57.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:57.727 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,703 [shard 0] osd - send_alive want=8 up_thru_wanted=0 2022-03-29T17:49:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - send_alive up_thru_wanted=8 up_thru=0 2022-03-29T17:49:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 NullEvt +create_info)): pg present 2022-03-29T17:49:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 NullEvt +create_info for pg: 1.0 2022-03-29T17:49:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - log is not dirty 2022-03-29T17:49:57.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:49:57.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,703 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:57.729 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,704 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e8 {1.0=8,2022-03-29T17:49:57.683718+0000}) v2): sub events complete 2022-03-29T17:49:57.729 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,704 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e8 {1.0=8,2022-03-29T17:49:57.683718+0000}) v2): complete 2022-03-29T17:49:57.730 INFO:tasks.ceph.osd.1.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/common/hobject.h:92:27: runtime error: left shift of negative value -1 2022-03-29T17:49:57.730 INFO:tasks.ceph.osd.1.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/common/hobject.h:92:57: runtime error: left shift of negative value -1 2022-03-29T17:49:57.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,704 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 IntervalFlush)): start 2022-03-29T17:49:57.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,704 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 IntervalFlush)): pg present 2022-03-29T17:49:57.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,704 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 IntervalFlush for pg: 1.0 2022-03-29T17:49:57.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,704 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:49:57.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,704 [shard 0] osd - log is not dirty 2022-03-29T17:49:57.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,704 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 IntervalFlush)): submitting ctx 2022-03-29T17:49:57.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,704 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:57.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,704 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:49:57.732 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,704 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.? v2:172.21.15.138:6804/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6804/35577, peer_name=osd.?, cc=333199852361566565 policy(lossy=false, server=false, standby=true, resetcheck=false) 2022-03-29T17:49:57.732 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,704 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.? v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:49:57.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,705 [shard 0] osd - pg_advance_map(id=1, detail=PGAdvanceMap(pg=1.0 from=8 to=8 do_init)): complete 2022-03-29T17:49:57.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,705 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> unknown.? -@61768] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:61768/0 2022-03-29T17:49:57.734 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,706 [shard 0] osd - send_alive want=8 up_thru_wanted=8 2022-03-29T17:49:57.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,706 [shard 0] osd - send_alive want=8 <= up_thru_wanted=8; skipping 2022-03-29T17:49:57.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,706 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 NullEvt +create_info)): complete 2022-03-29T17:49:57.735 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,706 [shard 0] osd - send_alive want=8 up_thru_wanted=8 2022-03-29T17:49:57.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,706 [shard 0] osd - send_alive want=8 <= up_thru_wanted=8; skipping 2022-03-29T17:49:57.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,706 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 IntervalFlush)): complete 2022-03-29T17:49:57.736 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,707 [shard 0] monc - get_auth_request(con=[osd.1(cluster) v2:172.21.15.138:6801/35578@61768 >> osd.? v2:172.21.15.138:6804/35577], auth_method=0) 2022-03-29T17:49:57.736 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,707 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.? -@61768] UPDATE: peer_type=osd, policy(lossy=false server=false standby=true resetcheck=false) 2022-03-29T17:49:57.748 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,747 [shard 0] monc - added challenge on [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.? -@61768] 2022-03-29T17:49:57.750 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:57,748 [shard 0] osd - handle_authentication osd.1 has caps allow * 2022-03-29T17:49:57.751 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,750 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.1 v2:172.21.15.138:6801/35578@61768] established: gs=0, pgs=1, cs=0, client_cookie=333199852361566565, server_cookie=13935239711636126743, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:57.752 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,750 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578@61768 >> osd.0 v2:172.21.15.138:6804/35577] connected: gs=1, pgs=1, cs=0, client_cookie=333199852361566565, server_cookie=13935239711636126743, in_seq=0, out_seq=0, out_q=1 2022-03-29T17:49:57.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:57,751 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:49:57.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:57,751 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MQuery 1.0 from 1 query_epoch 8 query: query(info 0'0 epoch_sent 8))): start 2022-03-29T17:49:57.754 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,751 [shard 0] osd - evt epoch is 8, i have 7, will wait 2022-03-29T17:49:57.754 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,751 [shard 0] osd - osdmap_subscribe(7) 2022-03-29T17:49:57.754 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,752 [shard 0] osd - handle_osd_map osd_map(8..8 src has 1..8) v4 2022-03-29T17:49:57.755 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,752 [shard 0] osd - handle_osd_map epochs [8..8], i have 7, src has [1..8] 2022-03-29T17:49:57.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:57,752 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:57.756 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,754 [shard 0] osd - osd.0: committed_osd_maps(8, 8) 2022-03-29T17:49:57.756 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,754 [shard 0] osd - osd.0: map e 8 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:49:57.757 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:57,754 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MQuery 1.0 from 1 query_epoch 8 query: query(info 0'0 epoch_sent 8))): got map 8 2022-03-29T17:49:57.757 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:57,755 [shard 0] osd - osd.0: now active 2022-03-29T17:49:57.758 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:49:57,755 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MQuery 1.0 from 1 query_epoch 8 query: query(info 0'0 epoch_sent 8))): pg absent, did not create 2022-03-29T17:49:57.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:57,755 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MQuery 1.0 from 1 query_epoch 8 query: query(info 0'0 epoch_sent 8))): OSDState is active 2022-03-29T17:49:57.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:57,755 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:49:57.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:57,755 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MQuery 1.0 from 1 query_epoch 8 query: query(info 0'0 epoch_sent 8))): complete 2022-03-29T17:49:57.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,792 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:49:57.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,792 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MNotifyRec 1.0 from 0 notify: (query:8 sent:8 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:49:57.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,792 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MNotifyRec 1.0 from 0 notify: (query:8 sent:8 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 8 2022-03-29T17:49:57.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,793 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MNotifyRec 1.0 from 0 notify: (query:8 sent:8 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:49:57.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,793 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 MNotifyRec 1.0 from 0 notify: (query:8 sent:8 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 1.0 2022-03-29T17:49:57.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,793 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering got osd.0 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:49:57.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,793 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:49:57.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,793 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:49:57.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,793 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:49:57.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,793 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:49:57.801 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,793 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.091114 4 2022-03-29T17:49:57.705383+0000 2022-03-29T17:49:57.801 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,793 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576197.702706, 1648576197.7053838 spent on 4 events 2022-03-29T17:49:57.801 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,793 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:49:57.802 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,793 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:49:57.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,793 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:49:57.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) 2022-03-29T17:49:57.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:49:57.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) 2022-03-29T17:49:57.803 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:49:57.803 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) 2022-03-29T17:49:57.803 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:49:57.803 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:49:57.804 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:49:57.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:49:57.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:49:57.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:49:57.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:49:57.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:49:57.805 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000470 0 0.000000 2022-03-29T17:49:57.806 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,794 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576197.7939124, 0.0 spent on 0 events 2022-03-29T17:49:57.806 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:49:57.806 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,794 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:49:57.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:49:57.806 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000068 0 0.000000 2022-03-29T17:49:57.807 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,794 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576197.7944322, 0.0 spent on 0 events 2022-03-29T17:49:57.807 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,794 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:49:57.807 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,794 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:49:57.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - log is not dirty 2022-03-29T17:49:57.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MNotifyRec 1.0 from 0 notify: (query:8 sent:8 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:49:57.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,794 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:57.808 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:57,795 [shard 0] osd - send_alive want=8 up_thru_wanted=8 2022-03-29T17:49:57.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,795 [shard 0] osd - send_alive want=8 <= up_thru_wanted=8; skipping 2022-03-29T17:49:57.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:57,795 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 MNotifyRec 1.0 from 0 notify: (query:8 sent:8 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:49:58.220 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,219 [shard 0] alienstore - stat 2022-03-29T17:49:58.221 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,220 [shard 0] alienstore - stat 2022-03-29T17:49:58.222 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:58,220 [shard 0] alienstore - stat 2022-03-29T17:49:58.696 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,694 [shard 0] osd - handle_osd_map osd_map(8..9 src has 1..9) v4 2022-03-29T17:49:58.696 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,694 [shard 0] osd - handle_osd_map epochs [8..9], i have 8, src has [1..9] 2022-03-29T17:49:58.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,695 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:58.697 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,695 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=1768554591214887716 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:58.697 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:58,696 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=673455304576882061 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:58.698 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:58,697 [shard 0] monc - get_auth_request(con=[osd.2(client) v2:172.21.15.138:6803/35579@61720 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:49:58.699 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,697 [shard 0] osd - osd.0: committed_osd_maps(9, 9) 2022-03-29T17:49:58.699 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,697 [shard 0] osd - osd.0: map e 9 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:49:58.700 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,698 [shard 0] monc - get_auth_request(con=[osd.0(client) v2:172.21.15.138:6805/35577@58956 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:49:58.700 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,698 [shard 0] osd - osd.0: now active 2022-03-29T17:49:58.701 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,698 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=3972459865308070716 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:49:58.701 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,699 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> unknown.? -@48748] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:48748/0 2022-03-29T17:49:58.702 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,700 [shard 0] osd - handle_osd_map osd_map(9..9 src has 1..9) v4 2022-03-29T17:49:58.702 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,700 [shard 0] osd - handle_osd_map epochs [9..9], i have 8, src has [1..9] 2022-03-29T17:49:58.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,701 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:58.703 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,701 [shard 0] monc - get_auth_request(con=[osd.1(client) v2:172.21.15.138:6800/35578@54668 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:49:58.703 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,701 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.? -@48748] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:49:58.704 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,701 [shard 0] monc - added challenge on [osd.1(client) v2:172.21.15.138:6800/35578 >> client.? -@48748] 2022-03-29T17:49:58.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,702 [shard 0] osd - handle_authentication mgr.x has caps allow * 2022-03-29T17:49:58.709 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,703 [shard 0] osd - osd.1: committed_osd_maps(9, 9) 2022-03-29T17:49:58.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,703 [shard 0] osd - osd.1: map e 9 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:49:58.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,703 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.4125 172.21.15.138:0/3250333293@48748] established: gs=0, pgs=3, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:58.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,703 [shard 0] osd - pg_advance_map(id=2, detail=PGAdvanceMap(pg=1.0 from=8 to=9)): start 2022-03-29T17:49:58.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,703 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:49:58.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:49:58.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 8, need_up_thru now false 2022-03-29T17:49:58.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:49:58.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:49:58.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:49:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:49:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 8:2 2022-03-29T17:49:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.909826 2 0.001530 2022-03-29T17:49:58.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,704 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576197.794544, 0.001530196 spent on 2 events 2022-03-29T17:49:58.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:49:58.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.001811 0 0.000000 2022-03-29T17:49:58.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,704 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576197.7026684, 0.0 spent on 0 events 2022-03-29T17:49:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:49:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,704 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:49:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:49:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:49:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:49:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 21.400600433s) 2022-03-29T17:49:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:49:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating activate peer osd.0 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:49:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:49:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,704 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:49:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating activate peer osd.0 1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:49:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:49:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:49:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:49:58.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:49:58.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:49:58.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:49:58.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:49:58.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:49:58.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:49:58.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:49:58.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:49:58.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:58.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 9:3 2022-03-29T17:49:58.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:49:58.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:49:58.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,705 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:49:58.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 8 while current is 9 2022-03-29T17:49:58.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,705 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,706 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4125.0:1 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:58.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,706 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4125.0:1 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:58.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,706 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:49:58.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,706 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=8 evt=epoch_sent: 9 epoch_requested: 8 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 21.400600433s int 16.000000000s) +create_info)): start 2022-03-29T17:49:58.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,707 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=8 evt=epoch_sent: 9 epoch_requested: 8 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 21.400600433s int 16.000000000s) +create_info)): got map 9 2022-03-29T17:49:58.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,707 [shard 0] osd - Creating 1.0 2022-03-29T17:49:58.727 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,707 [shard 0] osd - pg_epoch 8 pg[1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:49:58.727 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,707 [shard 0] osd - Entering state: Initial 2022-03-29T17:49:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,707 [shard 0] osd - pg_epoch 8 pg[1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:49:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,707 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:49:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,707 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:49:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,707 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:49:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:49:58.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 5.316920280s + 0.000000000s) 2022-03-29T17:49:58.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:58.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - pg_advance_map(id=1, detail=PGAdvanceMap(pg=1.0 from=8 to=9 do_init)): start 2022-03-29T17:49:58.729 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000762 0 0.000000 2022-03-29T17:49:58.730 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - Exiting state: Initial, entered at 1648576198.7075868, 0.0 spent on 0 events 2022-03-29T17:49:58.730 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:49:58.730 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - Entering state: Reset 2022-03-29T17:49:58.730 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 8 2022-03-29T17:49:58.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:49:58.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:49:58.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:49:58.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:49:58.731 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown exit Reset 0.000226 1 2022-03-29T17:49:58.708643+0000 2022-03-29T17:49:58.732 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - Exiting state: Reset, entered at 1648576198.7084508, 1648576198.7086437 spent on 1 events 2022-03-29T17:49:58.732 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:49:58.732 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - Entering state: Started 2022-03-29T17:49:58.732 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:49:58.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - Entering state: Start 2022-03-29T17:49:58.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:49:58.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown exit Start 0.000072 0 0.000000 2022-03-29T17:49:58.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - Exiting state: Start, entered at 1648576198.7087848, 0.0 spent on 0 events 2022-03-29T17:49:58.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:49:58.734 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,708 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:49:58.734 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 8 while current is 8 2022-03-29T17:49:58.734 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.734 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - pg_epoch 8 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:49:58.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:49:58.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:49:58.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:49:58.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 8 while current is 9 2022-03-29T17:49:58.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - Created 1.0 2022-03-29T17:49:58.736 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,709 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:49:58.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=8 evt=epoch_sent: 9 epoch_requested: 8 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 21.400600433s int 16.000000000s) +create_info)): pg present 2022-03-29T17:49:58.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 8 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 21.400600433s int 16.000000000s) +create_info for pg: 1.0 2022-03-29T17:49:58.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) log((0'0,0'0], crt=0'0) 2022-03-29T17:49:58.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,709 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:49:58.737 INFO:tasks.ceph.osd.0.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/common/hobject.h:92:27: runtime error: left shift of negative value -1 2022-03-29T17:49:58.738 INFO:tasks.ceph.osd.0.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/common/hobject.h:92:57: runtime error: left shift of negative value -1 2022-03-29T17:49:58.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9)): start 2022-03-29T17:49:58.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9)): pg present 2022-03-29T17:49:58.739 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,710 [shard 0] osd - evt epoch is 9, i have 8, will wait 2022-03-29T17:49:58.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:49:58.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 21.400600433s int 16.000000000s) 2022-03-29T17:49:58.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 21.319362640s 2022-03-29T17:49:58.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 21.319362640s 2022-03-29T17:49:58.743 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.001622 3 0.000406 2022-03-29T17:49:58.743 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,710 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576198.7089481, 0.000406105 spent on 3 events 2022-03-29T17:49:58.743 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:49:58.743 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,710 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:49:58.744 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:49:58.744 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,710 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:49:58.744 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:49:58.744 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:49:58.744 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,710 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:49:58.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=8 evt=epoch_sent: 9 epoch_requested: 8 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 21.400600433s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:49:58.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 IntervalFlush)): start 2022-03-29T17:49:58.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 IntervalFlush)): pg present 2022-03-29T17:49:58.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - pg_advance_map(id=1, detail=PGAdvanceMap(pg=1.0 from=8 to=9 do_init)): complete 2022-03-29T17:49:58.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 IntervalFlush for pg: 1.0 2022-03-29T17:49:58.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:49:58.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 IntervalFlush)): submitting ctx 2022-03-29T17:49:58.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,711 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.748 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,712 [shard 0] osd - osdmap_subscribe(8) 2022-03-29T17:49:58.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,713 [shard 0] osd - pg_advance_map(id=2, detail=PGAdvanceMap(pg=1.0 from=8 to=9)): complete 2022-03-29T17:49:58.749 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,713 [shard 0] osd - osd.1: now active 2022-03-29T17:49:58.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,713 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9 for pg: 1.0 2022-03-29T17:49:58.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,713 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 9 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 8 2022-03-29T17:49:58.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,713 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,713 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9)): submitting ctx 2022-03-29T17:49:58.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,713 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,713 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4125.0:1 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:58.751 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,713 [shard 0] osd - OpSequencer::start_op: op=1, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:49:58.751 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,714 [shard 0] osd - handle_osd_map osd_map(8..9 src has 1..9) v4 2022-03-29T17:49:58.751 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,714 [shard 0] osd - handle_osd_map epochs [8..9], i have 9, src has [1..9] 2022-03-29T17:49:58.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,714 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9)): start 2022-03-29T17:49:58.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9)): pg present 2022-03-29T17:49:58.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=8 evt=epoch_sent: 9 epoch_requested: 8 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 21.400600433s int 16.000000000s) +create_info)): complete 2022-03-29T17:49:58.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9 for pg: 1.0 2022-03-29T17:49:58.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 unknown state: react 9 telling primary 2022-03-29T17:49:58.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9)): submitting ctx 2022-03-29T17:49:58.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:49:58.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,715 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=0 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 IntervalFlush)): complete 2022-03-29T17:49:58.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,716 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9)): complete 2022-03-29T17:49:58.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,716 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 ActivateCommitted from 9 processed at 9)): complete 2022-03-29T17:49:58.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,747 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:49:58.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,747 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 0 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease_ack(ruub 21.400600433s))): start 2022-03-29T17:49:58.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 0 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease_ack(ruub 21.400600433s))): got map 9 2022-03-29T17:49:58.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 0 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease_ack(ruub 21.400600433s))): pg present 2022-03-29T17:49:58.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 9 MInfoRec from 0 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease_ack(ruub 21.400600433s) for pg: 1.0 2022-03-29T17:49:58.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 21.400600433s 2022-03-29T17:49:58.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 21.400600433s (sent 21.400600433s) 2022-03-29T17:49:58.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:49:58.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:49:58.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:49:58.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:49:58.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,748 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:49:58.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:49:58.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:49:58.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:49:58.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:49:58.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:58.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 9:4 2022-03-29T17:49:58.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:49:58.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 21.400600433s 2022-03-29T17:49:58.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 21.400600433s (sent 21.445096970s) 2022-03-29T17:49:58.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:49:58.775 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:49:58.775 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:49:58.775 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:49:58.775 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:49:58.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:49:58.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:49:58.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:49:58.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active state: mnow 5.445480824s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:49:58.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - send_pg_created 2022-03-29T17:49:58.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:49:58.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:49:58.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:49:58.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:49:58.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:49:58.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:49:58.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:49:58.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:49:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:49:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active publish_stats_to_osd 9:5 2022-03-29T17:49:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:49:58.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:49:58.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,749 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:49:58.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 AllReplicasRecovered)): start 2022-03-29T17:49:58.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:49:58.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:49:58.781 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:49:58.781 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:49:58.781 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:49:58.781 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:49:58.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:49:58.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:49:58.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:58.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active publish_stats_to_osd 9:6 2022-03-29T17:49:58.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 0 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease_ack(ruub 21.400600433s))): submitting ctx 2022-03-29T17:49:58.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,750 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4125.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {create, setxattr striper.version (16) in=31b, setxattr striper.excl in=12b, setxattr striper.size (16) in=28b, setxattr striper.allocated (16) in=33b, setxattr striper.layout.stripe_unit (16) in=42b, setxattr striper.layout.stripe_count (16) in=43b, setxattr striper.layout.object_size (16) in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:58.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - with_head_obc: cache miss on 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 AllReplicasRecovered)): pg present 2022-03-29T17:49:58.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 0 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease_ack(ruub 21.400600433s))): complete 2022-03-29T17:49:58.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - load_metadata: object 1:30306672:devicehealth::main.db.0000000000000000:head doesn't exist, returning empty metadata 2022-03-29T17:49:58.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - load_head_obc: loaded obs 1:30306672:devicehealth::main.db.0000000000000000:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - load_head_obc: returning obc 1:30306672:devicehealth::main.db.0000000000000000:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 9 AllReplicasRecovered for pg: 1.0 2022-03-29T17:49:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,751 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.046179 3 0.003457 2022-03-29T17:49:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,751 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576198.7055912, 0.003457726 spent on 3 events 2022-03-29T17:49:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,751 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:49:58.787 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,751 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:49:58.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:49:58.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:49:58.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:49:58.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,751 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:49:58.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:49:58.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:49:58.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:49:58.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:49:58.789 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:49:58.789 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:58.789 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active publish_stats_to_osd 9:7 2022-03-29T17:49:58.789 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000291 0 0.000000 2022-03-29T17:49:58.789 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,752 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576198.7518582, 0.0 spent on 0 events 2022-03-29T17:49:58.790 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:49:58.790 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,752 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:49:58.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:49:58.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:58.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 9:8 2022-03-29T17:49:58.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 AllReplicasRecovered)): submitting ctx 2022-03-29T17:49:58.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op create 2022-03-29T17:49:58.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - handling op create on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.793 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,752 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.793 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,752 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:58.793 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.793 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.793 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,753 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.794 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.794 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.794 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.794 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,753 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:58.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.795 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,753 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:58.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.796 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,753 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.layout.stripe_unit 2022-03-29T17:49:58.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.797 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,753 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.layout.stripe_count 2022-03-29T17:49:58.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.798 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,753 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.layout.object_size 2022-03-29T17:49:58.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:58.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:58.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:49:58.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {9'1 (0'0) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:1 2022-03-29T17:49:58.697728+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:58.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 9'1 (0'0) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:1 2022-03-29T17:49:58.697728+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:58.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=0'0 lua=0'0 crt=9'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:49:58.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=0'0 lua=0'0 crt=9'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:49:58.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:58.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,753 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:58.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,754 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:58.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,754 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 AllReplicasRecovered)): complete 2022-03-29T17:49:58.802 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:58,778 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@61720 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=2, pgs=6, cs=0, client_cookie=673455304576882061, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:58.802 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:58,779 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:49:58.803 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,780 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@58956 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=2, pgs=7, cs=0, client_cookie=1768554591214887716, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:58.803 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:58,781 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:49:58.804 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,782 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@54668 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=2, pgs=8, cs=0, client_cookie=3972459865308070716, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:49:58.804 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,783 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:49:58.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,788 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:49:58.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,788 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLease epoch 9 from osd.1 pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): start 2022-03-29T17:49:58.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,788 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLease epoch 9 from osd.1 pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): got map 9 2022-03-29T17:49:58.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,788 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLease epoch 9 from osd.1 pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): pg present 2022-03-29T17:49:58.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 9 MLease epoch 9 from osd.1 pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s) for pg: 1.0 2022-03-29T17:49:58.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s) 2022-03-29T17:49:58.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 21.398046494s 2022-03-29T17:49:58.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=0/0 les/c/f=0/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 21.398046494s 2022-03-29T17:49:58.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLease epoch 9 from osd.1 pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): submitting ctx 2022-03-29T17:49:58.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:49:58.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,789 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): start 2022-03-29T17:49:58.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): got map 9 2022-03-29T17:49:58.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLease epoch 9 from osd.1 pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): complete 2022-03-29T17:49:58.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): pg present 2022-03-29T17:49:58.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s) for pg: 1.0 2022-03-29T17:49:58.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 active update_history advanced history from ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8 2022-03-29T17:49:58.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): submitting ctx 2022-03-29T17:49:58.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:49:58.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,790 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): start 2022-03-29T17:49:58.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,791 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): got map 9 2022-03-29T17:49:58.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,791 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): pg present 2022-03-29T17:49:58.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,791 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s) for pg: 1.0 2022-03-29T17:49:58.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,791 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 active update_history advanced history from ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8 2022-03-29T17:49:58.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,791 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:49:58.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,791 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,791 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): submitting ctx 2022-03-29T17:49:58.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,791 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,792 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/0 les/c/f=9/0/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): complete 2022-03-29T17:49:58.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,792 [shard 0] osd - replicated_request(id=1, detail=RepRequest(from=1 req=osd_repop(client.4125.0:1 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'1, mlcod=9'1) v3)) start 2022-03-29T17:49:58.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,792 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MInfoRec from 1 info: 1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) pg_lease(ru 21.400600433s ub 21.445096970s int 16.000000000s))): complete 2022-03-29T17:49:58.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,792 [shard 0] osd - pg_epoch 9 pg[1.0( empty local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {9'1 (0'0) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:1 2022-03-29T17:49:58.697728+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:58.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,793 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 9'1 (0'0) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:1 2022-03-29T17:49:58.697728+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:58.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,793 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:49:58.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,793 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:49:58.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,793 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:58.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,793 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:58.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,793 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:58.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,829 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:49:58.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,829 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLeaseAck epoch 9 from osd.0 pg_lease_ack(ruub 21.445096970s))): start 2022-03-29T17:49:58.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,830 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLeaseAck epoch 9 from osd.0 pg_lease_ack(ruub 21.445096970s))): got map 9 2022-03-29T17:49:58.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,830 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLeaseAck epoch 9 from osd.0 pg_lease_ack(ruub 21.445096970s))): pg present 2022-03-29T17:49:58.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,830 [shard 0] osd - do_peering_event handling epoch_sent: 9 epoch_requested: 9 MLeaseAck epoch 9 from osd.0 pg_lease_ack(ruub 21.445096970s) for pg: 1.0 2022-03-29T17:49:58.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,830 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=0'0 lua=0'0 crt=9'1 lcod 0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 21.445096970s 2022-03-29T17:49:58.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,830 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=0'0 lua=0'0 crt=9'1 lcod 0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 21.445096970s (sent 21.445096970s) 2022-03-29T17:49:58.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,830 [shard 0] osd - log is not dirty 2022-03-29T17:49:58.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,830 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLeaseAck epoch 9 from osd.0 pg_lease_ack(ruub 21.445096970s))): submitting ctx 2022-03-29T17:49:58.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,830 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:58.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,831 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {create, setxattr striper.version (16) in=31b, setxattr striper.excl in=12b, setxattr striper.size (16) in=28b, setxattr striper.allocated (16) in=33b, setxattr striper.layout.stripe_unit (16) in=42b, setxattr striper.layout.stripe_count (16) in=43b, setxattr striper.layout.object_size (16) in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:58.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,831 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,831 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=0 pgid=1.0 sent=9 requested=9 evt=epoch_sent: 9 epoch_requested: 9 MLeaseAck epoch 9 from osd.0 pg_lease_ack(ruub 21.445096970s))): complete 2022-03-29T17:49:58.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,831 [shard 0] osd - OpSequencer::finish_op_in_order: op=1, last_started=1, last_unblocked=1, last_completed=0 2022-03-29T17:49:58.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4125.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {create, setxattr striper.version (16) in=31b, setxattr striper.excl in=12b, setxattr striper.size (16) in=28b, setxattr striper.allocated (16) in=33b, setxattr striper.layout.stripe_unit (16) in=42b, setxattr striper.layout.stripe_count (16) in=43b, setxattr striper.layout.object_size (16) in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:58.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4125.0:2 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:58.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4125.0:2 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:58.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4125.0:2 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:58.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - OpSequencer::start_op: op=2, last_started=1, last_unblocked=1, last_completed=1 2022-03-29T17:49:58.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4125.0:2 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:58.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:58.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,832 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:58.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:58.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:58.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:58.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,833 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:2 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:58.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - OpSequencer::finish_op_in_order: op=2, last_started=2, last_unblocked=2, last_completed=1 2022-03-29T17:49:58.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4125.0:2 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:58.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4125.0:3 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:58.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4125.0:3 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:58.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4125.0:3 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:58.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - OpSequencer::start_op: op=3, last_started=2, last_unblocked=2, last_completed=2 2022-03-29T17:49:58.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4125.0:3 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 0~100} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:58.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:58.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:49:58.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,834 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:58.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,835 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:3 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 0~100} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:58.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,835 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,835 [shard 0] osd - OpSequencer::finish_op_in_order: op=3, last_started=3, last_unblocked=3, last_completed=2 2022-03-29T17:49:58.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,835 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4125.0:3 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 0~100} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:58.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,835 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4125.0:4 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:58.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,835 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4125.0:4 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:58.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,835 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4125.0:4 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:58.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,835 [shard 0] osd - OpSequencer::start_op: op=4, last_started=3, last_unblocked=3, last_completed=3 2022-03-29T17:49:58.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4125.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:58.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:58.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:49:58.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:49:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,836 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.856 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,837 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:58.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:49:58.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:49:58.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:49:58.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.857 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,837 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:58.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:58.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'1 mlcod 9'1 active+clean calc_trim_to_aggressive limit = 9'1 2022-03-29T17:49:58.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'1 mlcod 9'1 active+clean append_log log((0'0,9'1], crt=9'1) {9'2 (9'1) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:4 2022-03-29T17:49:58.835513+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:58.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'1 lua=0'0 crt=9'1 lcod 9'1 mlcod 9'1 active+clean add_log_entry 9'2 (9'1) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:4 2022-03-29T17:49:58.835513+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:58.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'1 lua=0'0 crt=9'2 lcod 9'1 mlcod 9'1 active+clean append_log approx pg log length = 2 2022-03-29T17:49:58.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'1 lua=0'0 crt=9'2 lcod 9'1 mlcod 9'1 active+clean append_log transaction_applied = 1 2022-03-29T17:49:58.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:58.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,837 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:58.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,838 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:58.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,870 [shard 0] osd - replicated_request(id=2, detail=RepRequest(from=1 req=osd_repop(client.4125.0:4 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'2, mlcod=9'2) v3)) start 2022-03-29T17:49:58.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,871 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'1 (0'0,9'1] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'1 mlcod 9'1 active append_log log((0'0,9'1], crt=9'1) {9'2 (9'1) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:4 2022-03-29T17:49:58.835513+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:58.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,871 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'1 lcod 9'1 mlcod 9'1 active add_log_entry 9'2 (9'1) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:4 2022-03-29T17:49:58.835513+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:58.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,871 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'2 lcod 9'1 mlcod 9'1 active append_log approx pg log length = 2 2022-03-29T17:49:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,871 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'2 lcod 9'1 mlcod 9'1 active append_log transaction_applied = 1 2022-03-29T17:49:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,871 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,871 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,871 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:58.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,911 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:58.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,911 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,912 [shard 0] osd - OpSequencer::finish_op_in_order: op=4, last_started=4, last_unblocked=4, last_completed=3 2022-03-29T17:49:58.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,912 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4125.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:58.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,912 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4125.0:5 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:58.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,912 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4125.0:5 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:58.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4125.0:5 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:58.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - OpSequencer::start_op: op=5, last_started=4, last_unblocked=4, last_completed=4 2022-03-29T17:49:58.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4125.0:5 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:58.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:58.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,913 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:58.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,914 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,914 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,914 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:58.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,914 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:58.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,914 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:5 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:58.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,914 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,914 [shard 0] osd - OpSequencer::finish_op_in_order: op=5, last_started=5, last_unblocked=5, last_completed=4 2022-03-29T17:49:58.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,914 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4125.0:5 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:58.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4125.0:6 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:58.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4125.0:6 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:58.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4125.0:6 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:58.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - OpSequencer::start_op: op=6, last_started=5, last_unblocked=5, last_completed=5 2022-03-29T17:49:58.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:58.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4125.0:6 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:58.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:58.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - with_head_obc: cache miss on 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:58.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,915 [shard 0] osd - load_metadata: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head doesn't exist, returning empty metadata 2022-03-29T17:49:58.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - load_head_obc: loaded obs 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:58.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - load_head_obc: returning obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:58.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - handling op getxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:58.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - getxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - rollback_obc_if_modified: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:49:58.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:58.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - OpSequencer::finish_op_in_order: op=6, last_started=6, last_unblocked=6, last_completed=5 2022-03-29T17:49:58.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,916 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4125.0:6 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:58.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4125.0:7 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:58.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4125.0:7 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:58.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4125.0:7 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:58.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - OpSequencer::start_op: op=7, last_started=6, last_unblocked=6, last_completed=6 2022-03-29T17:49:58.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:58.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4125.0:7 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:58.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:58.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - with_head_obc: cache miss on 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:58.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - load_metadata: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head doesn't exist, returning empty metadata 2022-03-29T17:49:58.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - load_head_obc: loaded obs 1:29493a39:devicehealth::main.db-wal.0000000000000000:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:58.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - load_head_obc: returning obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:58.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:58.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:58.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,917 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:49:58.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:58.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - OpSequencer::finish_op_in_order: op=7, last_started=7, last_unblocked=7, last_completed=6 2022-03-29T17:49:58.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4125.0:7 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:58.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4125.0:8 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:58.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4125.0:8 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:58.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4125.0:8 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:58.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - OpSequencer::start_op: op=8, last_started=7, last_unblocked=7, last_completed=7 2022-03-29T17:49:58.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,918 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4125.0:8 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:58.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:58.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:49:58.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:49:58.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:58.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.941 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,919 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:58.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:49:58.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,919 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:49:58.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:58.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:58.943 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,920 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:58.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:58.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:49:58.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:58.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:58.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'2 mlcod 9'2 active+clean calc_trim_to_aggressive limit = 9'2 2022-03-29T17:49:58.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'2 mlcod 9'2 active+clean append_log log((0'0,9'2], crt=9'2) {9'3 (9'2) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:8 2022-03-29T17:49:58.918469+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:58.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'2 lua=0'0 crt=9'2 lcod 9'2 mlcod 9'2 active+clean add_log_entry 9'3 (9'2) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:8 2022-03-29T17:49:58.918469+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:58.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'2 lua=0'0 crt=9'3 lcod 9'2 mlcod 9'2 active+clean append_log approx pg log length = 3 2022-03-29T17:49:58.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'2 lua=0'0 crt=9'3 lcod 9'2 mlcod 9'2 active+clean append_log transaction_applied = 1 2022-03-29T17:49:58.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:58.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,920 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:58.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,921 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:58.954 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,952 [shard 0] osd - replicated_request(id=3, detail=RepRequest(from=1 req=osd_repop(client.4125.0:8 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'3, mlcod=9'3) v3)) start 2022-03-29T17:49:58.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,952 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'2 (0'0,9'2] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'2 mlcod 9'2 active append_log log((0'0,9'2], crt=9'2) {9'3 (9'2) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:8 2022-03-29T17:49:58.918469+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:58.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,953 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'2 lcod 9'2 mlcod 9'2 active add_log_entry 9'3 (9'2) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:8 2022-03-29T17:49:58.918469+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:58.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,953 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'3 lcod 9'2 mlcod 9'2 active append_log approx pg log length = 3 2022-03-29T17:49:58.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,953 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'3 lcod 9'2 mlcod 9'2 active append_log transaction_applied = 1 2022-03-29T17:49:58.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,953 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:58.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,953 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:58.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:58,953 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:58.995 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,993 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:8 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.000 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,994 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.000 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,994 [shard 0] osd - OpSequencer::finish_op_in_order: op=8, last_started=8, last_unblocked=8, last_completed=7 2022-03-29T17:49:59.000 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,994 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4125.0:8 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.001 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4125.0:9 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.001 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4125.0:9 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.001 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4125.0:9 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.001 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - OpSequencer::start_op: op=9, last_started=8, last_unblocked=8, last_completed=8 2022-03-29T17:49:59.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4125.0:9 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:49:59.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,995 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:49:59.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.004 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,996 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:49:59.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:49:59.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:49:59.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.006 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:58,996 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,996 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'3 mlcod 9'3 active+clean calc_trim_to_aggressive limit = 9'3 2022-03-29T17:49:59.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'3 mlcod 9'3 active+clean append_log log((0'0,9'3], crt=9'3) {9'4 (9'3) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:9 2022-03-29T17:49:58.994679+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'3 lua=0'0 crt=9'3 lcod 9'3 mlcod 9'3 active+clean add_log_entry 9'4 (9'3) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:9 2022-03-29T17:49:58.994679+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'3 lua=0'0 crt=9'4 lcod 9'3 mlcod 9'3 active+clean append_log approx pg log length = 4 2022-03-29T17:49:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=1 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'3 lua=0'0 crt=9'4 lcod 9'3 mlcod 9'3 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'4, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:58,997 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.036 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,034 [shard 0] osd - replicated_request(id=4, detail=RepRequest(from=1 req=osd_repop(client.4125.0:9 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'4, mlcod=9'4) v3)) start 2022-03-29T17:49:59.037 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,034 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'3 (0'0,9'3] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'3 mlcod 9'3 active append_log log((0'0,9'3], crt=9'3) {9'4 (9'3) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:9 2022-03-29T17:49:58.994679+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.037 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,035 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'3 lcod 9'3 mlcod 9'3 active add_log_entry 9'4 (9'3) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:9 2022-03-29T17:49:58.994679+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.037 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,035 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'4 lcod 9'3 mlcod 9'3 active append_log approx pg log length = 4 2022-03-29T17:49:59.037 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,035 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'4 lcod 9'3 mlcod 9'3 active append_log transaction_applied = 1 2022-03-29T17:49:59.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,035 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,035 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'4, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,035 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,075 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:9 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,076 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,076 [shard 0] osd - OpSequencer::finish_op_in_order: op=9, last_started=9, last_unblocked=9, last_completed=8 2022-03-29T17:49:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,076 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4125.0:9 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,076 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4125.0:10 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,076 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4125.0:10 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4125.0:10 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - OpSequencer::start_op: op=10, last_started=9, last_unblocked=9, last_completed=9 2022-03-29T17:49:59.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4125.0:10 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:59.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,077 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,078 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,078 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,078 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:10 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,078 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,078 [shard 0] osd - OpSequencer::finish_op_in_order: op=10, last_started=10, last_unblocked=10, last_completed=9 2022-03-29T17:49:59.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,078 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4125.0:10 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,078 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4125.0:11 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,078 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4125.0:11 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4125.0:11 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - OpSequencer::start_op: op=11, last_started=10, last_unblocked=10, last_completed=10 2022-03-29T17:49:59.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4125.0:11 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - handling op getxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - getxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - rollback_obc_if_modified: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:49:59.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,079 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - OpSequencer::finish_op_in_order: op=11, last_started=11, last_unblocked=11, last_completed=10 2022-03-29T17:49:59.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4125.0:11 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4125.0:12 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4125.0:12 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4125.0:12 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - OpSequencer::start_op: op=12, last_started=11, last_unblocked=11, last_completed=11 2022-03-29T17:49:59.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4125.0:12 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:49:59.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,080 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:12 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - OpSequencer::finish_op_in_order: op=12, last_started=12, last_unblocked=12, last_completed=11 2022-03-29T17:49:59.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4125.0:12 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4125.0:13 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4125.0:13 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4125.0:13 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - OpSequencer::start_op: op=13, last_started=12, last_unblocked=12, last_completed=12 2022-03-29T17:49:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4125.0:13 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,081 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:49:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,082 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,082 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:59.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,082 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,082 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:49:59.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,082 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:59.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,082 [shard 0] osd - OpSequencer::finish_op_in_order: op=13, last_started=13, last_unblocked=13, last_completed=12 2022-03-29T17:49:59.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,082 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4125.0:13 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4125.0:14 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4125.0:14 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4125.0:14 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - OpSequencer::start_op: op=14, last_started=13, last_unblocked=13, last_completed=13 2022-03-29T17:49:59.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4125.0:14 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {create, setxattr striper.version (16) in=31b, setxattr striper.excl in=12b, setxattr striper.size (16) in=28b, setxattr striper.allocated (16) in=33b, setxattr striper.layout.stripe_unit (16) in=42b, setxattr striper.layout.stripe_count (16) in=43b, setxattr striper.layout.object_size (16) in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op create 2022-03-29T17:49:59.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - handling op create on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.109 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,083 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.110 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,083 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,083 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.112 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,083 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:59.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,083 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.112 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,084 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.layout.stripe_unit 2022-03-29T17:49:59.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.113 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,084 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.layout.stripe_count 2022-03-29T17:49:59.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.114 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,084 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.layout.object_size 2022-03-29T17:49:59.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'4 mlcod 9'4 active+clean calc_trim_to_aggressive limit = 9'4 2022-03-29T17:49:59.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'4 mlcod 9'4 active+clean append_log log((0'0,9'4], crt=9'4) {9'5 (0'0) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:14 2022-03-29T17:49:59.082776+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'4 lua=0'0 crt=9'4 lcod 9'4 mlcod 9'4 active+clean add_log_entry 9'5 (0'0) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:14 2022-03-29T17:49:59.082776+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'4 lua=0'0 crt=9'5 lcod 9'4 mlcod 9'4 active+clean append_log approx pg log length = 5 2022-03-29T17:49:59.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'4 lua=0'0 crt=9'5 lcod 9'4 mlcod 9'4 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'5, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,084 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.120 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,117 [shard 0] osd - replicated_request(id=5, detail=RepRequest(from=1 req=osd_repop(client.4125.0:14 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'5, mlcod=9'5) v3)) start 2022-03-29T17:49:59.120 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,118 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'4 (0'0,9'4] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'4 mlcod 9'4 active append_log log((0'0,9'4], crt=9'4) {9'5 (0'0) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:14 2022-03-29T17:49:59.082776+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.120 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,118 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'4 lcod 9'4 mlcod 9'4 active add_log_entry 9'5 (0'0) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:14 2022-03-29T17:49:59.082776+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.120 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,118 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'5 lcod 9'4 mlcod 9'4 active append_log approx pg log length = 5 2022-03-29T17:49:59.121 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,118 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'5 lcod 9'4 mlcod 9'4 active append_log transaction_applied = 1 2022-03-29T17:49:59.121 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,118 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.121 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,118 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'5, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.121 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,118 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,159 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:14 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {create, setxattr striper.version (16) in=31b, setxattr striper.excl in=12b, setxattr striper.size (16) in=28b, setxattr striper.allocated (16) in=33b, setxattr striper.layout.stripe_unit (16) in=42b, setxattr striper.layout.stripe_count (16) in=43b, setxattr striper.layout.object_size (16) in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,159 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,159 [shard 0] osd - OpSequencer::finish_op_in_order: op=14, last_started=14, last_unblocked=14, last_completed=13 2022-03-29T17:49:59.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,159 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4125.0:14 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {create, setxattr striper.version (16) in=31b, setxattr striper.excl in=12b, setxattr striper.size (16) in=28b, setxattr striper.allocated (16) in=33b, setxattr striper.layout.stripe_unit (16) in=42b, setxattr striper.layout.stripe_count (16) in=43b, setxattr striper.layout.object_size (16) in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4125.0:15 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4125.0:15 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4125.0:15 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - OpSequencer::start_op: op=15, last_started=14, last_unblocked=14, last_completed=14 2022-03-29T17:49:59.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4125.0:15 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - handling op getxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - getxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - handling op getxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,160 [shard 0] osd - getxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - handling op getxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - getxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:59.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - handling op getxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - getxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:15 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - OpSequencer::finish_op_in_order: op=15, last_started=15, last_unblocked=15, last_completed=14 2022-03-29T17:49:59.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,161 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4125.0:15 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4125.0:16 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4125.0:16 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4125.0:16 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - OpSequencer::start_op: op=16, last_started=15, last_unblocked=15, last_completed=15 2022-03-29T17:49:59.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4125.0:16 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.allocated (16) in=33b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.175 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,162 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:59.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.176 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,162 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,162 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'5 mlcod 9'5 active+clean calc_trim_to_aggressive limit = 9'5 2022-03-29T17:49:59.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,163 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'5 mlcod 9'5 active+clean append_log log((0'0,9'5], crt=9'5) {9'6 (9'5) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:16 2022-03-29T17:49:59.161854+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,163 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'5 lua=0'0 crt=9'5 lcod 9'5 mlcod 9'5 active+clean add_log_entry 9'6 (9'5) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:16 2022-03-29T17:49:59.161854+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,163 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'5 lua=0'0 crt=9'6 lcod 9'5 mlcod 9'5 active+clean append_log approx pg log length = 6 2022-03-29T17:49:59.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,163 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'5 lua=0'0 crt=9'6 lcod 9'5 mlcod 9'5 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,163 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,163 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'6, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,163 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.202 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,199 [shard 0] osd - replicated_request(id=6, detail=RepRequest(from=1 req=osd_repop(client.4125.0:16 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'6, mlcod=9'6) v3)) start 2022-03-29T17:49:59.202 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,200 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'5 (0'0,9'5] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'5 mlcod 9'5 active append_log log((0'0,9'5], crt=9'5) {9'6 (9'5) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:16 2022-03-29T17:49:59.161854+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.202 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,200 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'5 lcod 9'5 mlcod 9'5 active add_log_entry 9'6 (9'5) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:16 2022-03-29T17:49:59.161854+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.202 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,200 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'6 lcod 9'5 mlcod 9'5 active append_log approx pg log length = 6 2022-03-29T17:49:59.203 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,200 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'6 lcod 9'5 mlcod 9'5 active append_log transaction_applied = 1 2022-03-29T17:49:59.203 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,200 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.203 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,200 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'6, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.203 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,200 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.228 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:59,226 [shard 0] alienstore - stat 2022-03-29T17:49:59.228 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:59,227 [shard 0] alienstore - stat 2022-03-29T17:49:59.229 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,227 [shard 0] alienstore - stat 2022-03-29T17:49:59.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,242 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:16 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.allocated (16) in=33b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,242 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,242 [shard 0] osd - OpSequencer::finish_op_in_order: op=16, last_started=16, last_unblocked=16, last_completed=15 2022-03-29T17:49:59.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,242 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4125.0:16 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.allocated (16) in=33b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4125.0:17 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4125.0:17 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4125.0:17 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - OpSequencer::start_op: op=17, last_started=16, last_unblocked=16, last_completed=16 2022-03-29T17:49:59.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4125.0:17 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,243 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'6 mlcod 9'6 active+clean calc_trim_to_aggressive limit = 9'6 2022-03-29T17:49:59.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'6 mlcod 9'6 active+clean append_log log((0'0,9'6], crt=9'6) {9'7 (9'6) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:17 2022-03-29T17:49:59.242689+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'6 lua=0'0 crt=9'6 lcod 9'6 mlcod 9'6 active+clean add_log_entry 9'7 (9'6) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:17 2022-03-29T17:49:59.242689+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'6 lua=0'0 crt=9'7 lcod 9'6 mlcod 9'6 active+clean append_log approx pg log length = 7 2022-03-29T17:49:59.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'6 lua=0'0 crt=9'7 lcod 9'6 mlcod 9'6 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'7, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4125.0:18 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,244 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4125.0:18 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4125.0:18 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - OpSequencer::start_op: op=18, last_started=17, last_unblocked=17, last_completed=16 2022-03-29T17:49:59.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4125.0:18 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.259 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,245 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:59.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.260 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,245 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'6 lua=0'0 crt=9'7 lcod 9'6 mlcod 9'6 active+clean calc_trim_to_aggressive limit = 9'6 2022-03-29T17:49:59.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,245 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'6 lua=0'0 crt=9'7 lcod 9'6 mlcod 9'6 active+clean append_log log((0'0,9'7], crt=9'7) {9'8 (9'7) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:18 2022-03-29T17:49:59.242898+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,246 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'6 lua=0'0 crt=9'7 lcod 9'6 mlcod 9'6 active+clean add_log_entry 9'8 (9'7) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:18 2022-03-29T17:49:59.242898+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,246 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'6 lua=0'0 crt=9'8 lcod 9'6 mlcod 9'6 active+clean append_log approx pg log length = 8 2022-03-29T17:49:59.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,246 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'6 lua=0'0 crt=9'8 lcod 9'6 mlcod 9'6 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,246 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,246 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'8, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,246 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,282 [shard 0] osd - replicated_request(id=7, detail=RepRequest(from=1 req=osd_repop(client.4125.0:17 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'7, mlcod=9'7) v3)) start 2022-03-29T17:49:59.285 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,283 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'6 (0'0,9'6] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'6 mlcod 9'6 active append_log log((0'0,9'6], crt=9'6) {9'7 (9'6) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:17 2022-03-29T17:49:59.242689+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.285 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,283 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'6 lcod 9'6 mlcod 9'6 active add_log_entry 9'7 (9'6) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:17 2022-03-29T17:49:59.242689+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.286 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,283 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'7 lcod 9'6 mlcod 9'6 active append_log approx pg log length = 7 2022-03-29T17:49:59.286 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,283 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'7 lcod 9'6 mlcod 9'6 active append_log transaction_applied = 1 2022-03-29T17:49:59.286 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,283 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.286 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,283 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'7, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.287 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,283 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.287 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,283 [shard 0] osd - replicated_request(id=8, detail=RepRequest(from=1 req=osd_repop(client.4125.0:18 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'8, mlcod=9'8) v3)) start 2022-03-29T17:49:59.288 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,285 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'7 (0'0,9'7] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'7 mlcod 9'7 active append_log log((0'0,9'7], crt=9'7) {9'8 (9'7) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:18 2022-03-29T17:49:59.242898+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.288 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,285 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'7 lcod 9'7 mlcod 9'7 active add_log_entry 9'8 (9'7) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:18 2022-03-29T17:49:59.242898+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.288 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,285 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'8 lcod 9'7 mlcod 9'7 active append_log approx pg log length = 8 2022-03-29T17:49:59.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,285 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'8 lcod 9'7 mlcod 9'7 active append_log transaction_applied = 1 2022-03-29T17:49:59.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,285 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,285 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'8, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,285 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,324 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:17 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,324 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,324 [shard 0] osd - OpSequencer::finish_op_in_order: op=17, last_started=18, last_unblocked=18, last_completed=16 2022-03-29T17:49:59.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,324 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4125.0:17 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,325 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:18 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,325 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,325 [shard 0] osd - OpSequencer::finish_op_in_order: op=18, last_started=18, last_unblocked=18, last_completed=17 2022-03-29T17:49:59.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,325 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4125.0:18 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,365 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4125.0:19 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,365 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4125.0:19 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,365 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4125.0:19 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,365 [shard 0] osd - OpSequencer::start_op: op=19, last_started=18, last_unblocked=18, last_completed=18 2022-03-29T17:49:59.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4125.0:19 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.allocated (16) in=33b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.375 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,366 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:59.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,366 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'8 mlcod 9'8 active+clean calc_trim_to_aggressive limit = 9'8 2022-03-29T17:49:59.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'8 mlcod 9'8 active+clean append_log log((0'0,9'8], crt=9'8) {9'9 (9'4) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:19 2022-03-29T17:49:59.365433+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'8 lua=0'0 crt=9'8 lcod 9'8 mlcod 9'8 active+clean add_log_entry 9'9 (9'4) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:19 2022-03-29T17:49:59.365433+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'8 lua=0'0 crt=9'9 lcod 9'8 mlcod 9'8 active+clean append_log approx pg log length = 9 2022-03-29T17:49:59.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'8 lua=0'0 crt=9'9 lcod 9'8 mlcod 9'8 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,366 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'9, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,367 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,368 [shard 0] osd - replicated_request(id=9, detail=RepRequest(from=1 req=osd_repop(client.4125.0:19 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'9, mlcod=9'9) v3)) start 2022-03-29T17:49:59.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,368 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'8 (0'0,9'8] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'8 mlcod 9'8 active append_log log((0'0,9'8], crt=9'8) {9'9 (9'4) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:19 2022-03-29T17:49:59.365433+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,368 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'8 lcod 9'8 mlcod 9'8 active add_log_entry 9'9 (9'4) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:19 2022-03-29T17:49:59.365433+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,368 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'9 lcod 9'8 mlcod 9'8 active append_log approx pg log length = 9 2022-03-29T17:49:59.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,368 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'9 lcod 9'8 mlcod 9'8 active append_log transaction_applied = 1 2022-03-29T17:49:59.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,368 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,368 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'9, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,368 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,409 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:19 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.allocated (16) in=33b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,410 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,410 [shard 0] osd - OpSequencer::finish_op_in_order: op=19, last_started=19, last_unblocked=19, last_completed=18 2022-03-29T17:49:59.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,410 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4125.0:19 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.allocated (16) in=33b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,411 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4125.0:20 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,411 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4125.0:20 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,411 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4125.0:20 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,411 [shard 0] osd - OpSequencer::start_op: op=20, last_started=19, last_unblocked=19, last_completed=19 2022-03-29T17:49:59.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,411 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,411 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4125.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,411 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:49:59.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'9 mlcod 9'9 active+clean calc_trim_to_aggressive limit = 9'9 2022-03-29T17:49:59.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'9 mlcod 9'9 active+clean append_log log((0'0,9'9], crt=9'9) {9'10 (9'9) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:20 2022-03-29T17:49:59.410534+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'9 lcod 9'9 mlcod 9'9 active+clean add_log_entry 9'10 (9'9) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:20 2022-03-29T17:49:59.410534+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'10 lcod 9'9 mlcod 9'9 active+clean append_log approx pg log length = 10 2022-03-29T17:49:59.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'10 lcod 9'9 mlcod 9'9 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'10, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,412 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,413 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4125.0:21 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,413 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4125.0:21 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.421 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,414 [shard 0] osd - replicated_request(id=10, detail=RepRequest(from=1 req=osd_repop(client.4125.0:20 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'10, mlcod=9'10) v3)) start 2022-03-29T17:49:59.421 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,414 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'9 (0'0,9'9] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'9 mlcod 9'9 active append_log log((0'0,9'9], crt=9'9) {9'10 (9'9) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:20 2022-03-29T17:49:59.410534+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.421 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,414 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'9 lcod 9'9 mlcod 9'9 active add_log_entry 9'10 (9'9) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:20 2022-03-29T17:49:59.410534+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,414 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'10 lcod 9'9 mlcod 9'9 active append_log approx pg log length = 10 2022-03-29T17:49:59.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,414 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'10 lcod 9'9 mlcod 9'9 active append_log transaction_applied = 1 2022-03-29T17:49:59.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,414 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,414 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'10, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,414 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,415 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4125.0:21 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,415 [shard 0] osd - OpSequencer::start_op: op=21, last_started=20, last_unblocked=20, last_completed=19 2022-03-29T17:49:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,416 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,416 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4125.0:22 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,416 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4125.0:22 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,416 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4125.0:22 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,416 [shard 0] osd - OpSequencer::start_op: op=22, last_started=21, last_unblocked=21, last_completed=19 2022-03-29T17:49:59.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,416 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4125.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,416 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,416 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:49:59.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'10 lcod 9'9 mlcod 9'9 active+clean calc_trim_to_aggressive limit = 9'9 2022-03-29T17:49:59.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'10 lcod 9'9 mlcod 9'9 active+clean append_log log((0'0,9'10], crt=9'10) {9'11 (9'10) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:21 2022-03-29T17:49:59.410645+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'10 lcod 9'9 mlcod 9'9 active+clean add_log_entry 9'11 (9'10) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:21 2022-03-29T17:49:59.410645+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'11 lcod 9'9 mlcod 9'9 active+clean append_log approx pg log length = 11 2022-03-29T17:49:59.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'11 lcod 9'9 mlcod 9'9 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'11, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,417 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.431 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,418 [shard 0] osd - replicated_request(id=11, detail=RepRequest(from=1 req=osd_repop(client.4125.0:21 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'11, mlcod=9'11) v3)) start 2022-03-29T17:49:59.431 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,419 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'10 (0'0,9'10] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'10 mlcod 9'10 active append_log log((0'0,9'10], crt=9'10) {9'11 (9'10) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:21 2022-03-29T17:49:59.410645+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.432 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,419 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'10 lcod 9'10 mlcod 9'10 active add_log_entry 9'11 (9'10) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:21 2022-03-29T17:49:59.410645+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.432 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,419 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'11 lcod 9'10 mlcod 9'10 active append_log approx pg log length = 11 2022-03-29T17:49:59.432 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,419 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'11 lcod 9'10 mlcod 9'10 active append_log transaction_applied = 1 2022-03-29T17:49:59.432 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,419 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.433 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,419 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'11, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.433 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,419 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,420 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,420 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4125.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,420 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,420 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,420 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,420 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.437 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,421 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:59.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.438 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,421 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'11 lcod 9'9 mlcod 9'9 active+clean calc_trim_to_aggressive limit = 9'9 2022-03-29T17:49:59.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'11 lcod 9'9 mlcod 9'9 active+clean append_log log((0'0,9'11], crt=9'11) {9'12 (9'11) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:22 2022-03-29T17:49:59.410689+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'11 lcod 9'9 mlcod 9'9 active+clean add_log_entry 9'12 (9'11) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:22 2022-03-29T17:49:59.410689+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'12 lcod 9'9 mlcod 9'9 active+clean append_log approx pg log length = 12 2022-03-29T17:49:59.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'9 lua=0'0 crt=9'12 lcod 9'9 mlcod 9'9 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'12, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,421 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.441 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,422 [shard 0] osd - replicated_request(id=12, detail=RepRequest(from=1 req=osd_repop(client.4125.0:22 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'12, mlcod=9'12) v3)) start 2022-03-29T17:49:59.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,423 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'11 (0'0,9'11] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'11 mlcod 9'11 active append_log log((0'0,9'11], crt=9'11) {9'12 (9'11) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:22 2022-03-29T17:49:59.410689+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,423 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'11 lcod 9'11 mlcod 9'11 active add_log_entry 9'12 (9'11) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:22 2022-03-29T17:49:59.410689+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,423 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'12 lcod 9'11 mlcod 9'11 active append_log approx pg log length = 12 2022-03-29T17:49:59.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,423 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'12 lcod 9'11 mlcod 9'11 active append_log transaction_applied = 1 2022-03-29T17:49:59.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,423 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,423 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'12, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,423 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,424 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,425 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,425 [shard 0] osd - OpSequencer::finish_op_in_order: op=20, last_started=22, last_unblocked=22, last_completed=19 2022-03-29T17:49:59.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,425 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4125.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,425 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,426 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,426 [shard 0] osd - OpSequencer::finish_op_in_order: op=21, last_started=22, last_unblocked=22, last_completed=20 2022-03-29T17:49:59.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,426 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4125.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,427 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,427 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,427 [shard 0] osd - OpSequencer::finish_op_in_order: op=22, last_started=22, last_unblocked=22, last_completed=21 2022-03-29T17:49:59.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,427 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4125.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.468 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,465 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4125.0:23 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.469 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,465 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4125.0:23 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.469 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,465 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4125.0:23 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.469 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,465 [shard 0] osd - OpSequencer::start_op: op=23, last_started=22, last_unblocked=22, last_completed=22 2022-03-29T17:49:59.469 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4125.0:23 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'12 mlcod 9'12 active+clean calc_trim_to_aggressive limit = 9'12 2022-03-29T17:49:59.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'12 mlcod 9'12 active+clean append_log log((0'0,9'12], crt=9'12) {9'13 (9'8) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:23 2022-03-29T17:49:59.465408+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.472 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'12 lua=0'0 crt=9'12 lcod 9'12 mlcod 9'12 active+clean add_log_entry 9'13 (9'8) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:23 2022-03-29T17:49:59.465408+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.472 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'12 lua=0'0 crt=9'13 lcod 9'12 mlcod 9'12 active+clean append_log approx pg log length = 13 2022-03-29T17:49:59.472 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'12 lua=0'0 crt=9'13 lcod 9'12 mlcod 9'12 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.472 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.473 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'13, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.473 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,466 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,468 [shard 0] osd - replicated_request(id=13, detail=RepRequest(from=1 req=osd_repop(client.4125.0:23 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'13, mlcod=9'13) v3)) start 2022-03-29T17:49:59.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,468 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'12 (0'0,9'12] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'12 mlcod 9'12 active append_log log((0'0,9'12], crt=9'12) {9'13 (9'8) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:23 2022-03-29T17:49:59.465408+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,468 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'12 lcod 9'12 mlcod 9'12 active add_log_entry 9'13 (9'8) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:23 2022-03-29T17:49:59.465408+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,468 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'13 lcod 9'12 mlcod 9'12 active append_log approx pg log length = 13 2022-03-29T17:49:59.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,468 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'13 lcod 9'12 mlcod 9'12 active append_log transaction_applied = 1 2022-03-29T17:49:59.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,468 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.476 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,468 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'13, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.476 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,468 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,509 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:23 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,509 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,509 [shard 0] osd - OpSequencer::finish_op_in_order: op=23, last_started=23, last_unblocked=23, last_completed=22 2022-03-29T17:49:59.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,509 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4125.0:23 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.515 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4125.0:24 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4125.0:24 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4125.0:24 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - OpSequencer::start_op: op=24, last_started=23, last_unblocked=23, last_completed=23 2022-03-29T17:49:59.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4125.0:24 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:49:59.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,510 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:49:59.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.519 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,511 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:49:59.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:49:59.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,511 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.521 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,512 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:49:59.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'13 mlcod 9'13 active+clean calc_trim_to_aggressive limit = 9'13 2022-03-29T17:49:59.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'13 mlcod 9'13 active+clean append_log log((0'0,9'13], crt=9'13) {9'14 (9'12) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:24 2022-03-29T17:49:59.509796+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'13 lua=0'0 crt=9'13 lcod 9'13 mlcod 9'13 active+clean add_log_entry 9'14 (9'12) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:24 2022-03-29T17:49:59.509796+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'13 lua=0'0 crt=9'14 lcod 9'13 mlcod 9'13 active+clean append_log approx pg log length = 14 2022-03-29T17:49:59.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'13 lua=0'0 crt=9'14 lcod 9'13 mlcod 9'13 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'14, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,512 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,549 [shard 0] osd - replicated_request(id=14, detail=RepRequest(from=1 req=osd_repop(client.4125.0:24 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'14, mlcod=9'14) v3)) start 2022-03-29T17:49:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,550 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'13 (0'0,9'13] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'13 mlcod 9'13 active append_log log((0'0,9'13], crt=9'13) {9'14 (9'12) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:24 2022-03-29T17:49:59.509796+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,550 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'13 lcod 9'13 mlcod 9'13 active add_log_entry 9'14 (9'12) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:24 2022-03-29T17:49:59.509796+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,550 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'14 lcod 9'13 mlcod 9'13 active append_log approx pg log length = 14 2022-03-29T17:49:59.553 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,550 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'14 lcod 9'13 mlcod 9'13 active append_log transaction_applied = 1 2022-03-29T17:49:59.553 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,550 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.553 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,550 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'14, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.553 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,550 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.581 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-03-29T17:49:59.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,590 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:24 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,591 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,591 [shard 0] osd - OpSequencer::finish_op_in_order: op=24, last_started=24, last_unblocked=24, last_completed=23 2022-03-29T17:49:59.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,591 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4125.0:24 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4125.0:25 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4125.0:25 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4125.0:25 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - OpSequencer::start_op: op=25, last_started=24, last_unblocked=24, last_completed=24 2022-03-29T17:49:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4125.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:49:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,592 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:49:59.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,593 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,593 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,593 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.601 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,593 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,593 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,593 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:49:59.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,593 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:49:59.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,593 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,593 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:49:59.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.603 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,594 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'14 mlcod 9'14 active+clean calc_trim_to_aggressive limit = 9'14 2022-03-29T17:49:59.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'14 mlcod 9'14 active+clean append_log log((0'0,9'14], crt=9'14) {9'15 (9'14) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:25 2022-03-29T17:49:59.591804+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'14 lua=0'0 crt=9'14 lcod 9'14 mlcod 9'14 active+clean add_log_entry 9'15 (9'14) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:25 2022-03-29T17:49:59.591804+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'14 lua=0'0 crt=9'15 lcod 9'14 mlcod 9'14 active+clean append_log approx pg log length = 15 2022-03-29T17:49:59.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'14 lua=0'0 crt=9'15 lcod 9'14 mlcod 9'14 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'15, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,594 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.634 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,632 [shard 0] osd - replicated_request(id=15, detail=RepRequest(from=1 req=osd_repop(client.4125.0:25 1.0 e9/8 1:30306672:devicehealth::main.db.0000000000000000:head v 9'15, mlcod=9'15) v3)) start 2022-03-29T17:49:59.634 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,633 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'14 (0'0,9'14] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'14 mlcod 9'14 active append_log log((0'0,9'14], crt=9'14) {9'15 (9'14) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:25 2022-03-29T17:49:59.591804+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.635 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,633 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'14 lcod 9'14 mlcod 9'14 active add_log_entry 9'15 (9'14) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:25 2022-03-29T17:49:59.591804+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,633 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'15 lcod 9'14 mlcod 9'14 active append_log approx pg log length = 15 2022-03-29T17:49:59.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,633 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'15 lcod 9'14 mlcod 9'14 active append_log transaction_applied = 1 2022-03-29T17:49:59.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,633 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,633 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'15, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.637 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,634 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,673 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,674 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,674 [shard 0] osd - OpSequencer::finish_op_in_order: op=25, last_started=25, last_unblocked=25, last_completed=24 2022-03-29T17:49:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,674 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4125.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,674 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4125.0:26 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,674 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4125.0:26 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4125.0:26 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - OpSequencer::start_op: op=26, last_started=25, last_unblocked=25, last_completed=25 2022-03-29T17:49:59.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4125.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,675 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:59.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:49:59.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,676 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - OpSequencer::finish_op_in_order: op=26, last_started=26, last_unblocked=26, last_completed=25 2022-03-29T17:49:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4125.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4125.0:27 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4125.0:27 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4125.0:27 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - OpSequencer::start_op: op=27, last_started=26, last_unblocked=26, last_completed=26 2022-03-29T17:49:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4125.0:27 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op read 2022-03-29T17:49:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,677 [shard 0] osd - handling op read on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - read: data length: 1 2022-03-29T17:49:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:27 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - OpSequencer::finish_op_in_order: op=27, last_started=27, last_unblocked=27, last_completed=26 2022-03-29T17:49:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4125.0:27 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4125.0:28 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4125.0:28 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4125.0:28 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - OpSequencer::start_op: op=28, last_started=27, last_unblocked=27, last_completed=27 2022-03-29T17:49:59.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4125.0:28 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,678 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:49:59.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - read: data length: 16 2022-03-29T17:49:59.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:28 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - OpSequencer::finish_op_in_order: op=28, last_started=28, last_unblocked=28, last_completed=27 2022-03-29T17:49:59.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,679 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4125.0:28 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4125.0:29 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4125.0:29 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4125.0:29 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - OpSequencer::start_op: op=29, last_started=28, last_unblocked=28, last_completed=28 2022-03-29T17:49:59.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:59.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4125.0:29 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:59.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:49:59.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:49:59.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:59.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:49:59.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,680 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:49:59.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - OpSequencer::finish_op_in_order: op=29, last_started=29, last_unblocked=29, last_completed=28 2022-03-29T17:49:59.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4125.0:29 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4125.0:30 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4125.0:30 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4125.0:30 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - OpSequencer::start_op: op=30, last_started=29, last_unblocked=29, last_completed=29 2022-03-29T17:49:59.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4125.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,681 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'15 mlcod 9'15 active+clean calc_trim_to_aggressive limit = 9'15 2022-03-29T17:49:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'15 mlcod 9'15 active+clean append_log log((0'0,9'15], crt=9'15) {9'16 (9'13) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:30 2022-03-29T17:49:59.681062+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'15 lcod 9'15 mlcod 9'15 active+clean add_log_entry 9'16 (9'13) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:30 2022-03-29T17:49:59.681062+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'16 lcod 9'15 mlcod 9'15 active+clean append_log approx pg log length = 16 2022-03-29T17:49:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'16 lcod 9'15 mlcod 9'15 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'16, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4125.0:31 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,682 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4125.0:31 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4125.0:31 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - OpSequencer::start_op: op=31, last_started=30, last_unblocked=30, last_completed=29 2022-03-29T17:49:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4125.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,683 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,684 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'16 lcod 9'15 mlcod 9'15 active+clean calc_trim_to_aggressive limit = 9'15 2022-03-29T17:49:59.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,684 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'16 lcod 9'15 mlcod 9'15 active+clean append_log log((0'0,9'16], crt=9'16) {9'17 (9'16) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:31 2022-03-29T17:49:59.681098+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,684 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'16 lcod 9'15 mlcod 9'15 active+clean add_log_entry 9'17 (9'16) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:31 2022-03-29T17:49:59.681098+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,684 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'17 lcod 9'15 mlcod 9'15 active+clean append_log approx pg log length = 17 2022-03-29T17:49:59.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,684 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'17 lcod 9'15 mlcod 9'15 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,684 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,684 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'17, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,684 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4125.0:32 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4125.0:32 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4125.0:32 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - OpSequencer::start_op: op=32, last_started=31, last_unblocked=31, last_completed=29 2022-03-29T17:49:59.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4125.0:33 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4125.0:33 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4125.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,685 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'17 lcod 9'15 mlcod 9'15 active+clean calc_trim_to_aggressive limit = 9'15 2022-03-29T17:49:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'17 lcod 9'15 mlcod 9'15 active+clean append_log log((0'0,9'17], crt=9'17) {9'18 (9'17) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:32 2022-03-29T17:49:59.681132+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'17 lcod 9'15 mlcod 9'15 active+clean add_log_entry 9'18 (9'17) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:32 2022-03-29T17:49:59.681132+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'18 lcod 9'15 mlcod 9'15 active+clean append_log approx pg log length = 18 2022-03-29T17:49:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'18 lcod 9'15 mlcod 9'15 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'18, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4125.0:33 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,686 [shard 0] osd - OpSequencer::start_op: op=33, last_started=32, last_unblocked=32, last_completed=29 2022-03-29T17:49:59.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,687 [shard 0] osd - replicated_request(id=16, detail=RepRequest(from=1 req=osd_repop(client.4125.0:30 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'16, mlcod=9'16) v3)) start 2022-03-29T17:49:59.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,687 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'15 (0'0,9'15] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'15 mlcod 9'15 active append_log log((0'0,9'15], crt=9'15) {9'16 (9'13) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:30 2022-03-29T17:49:59.681062+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,687 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'15 lcod 9'15 mlcod 9'15 active add_log_entry 9'16 (9'13) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:30 2022-03-29T17:49:59.681062+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,687 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'16 lcod 9'15 mlcod 9'15 active append_log approx pg log length = 16 2022-03-29T17:49:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,687 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'16 lcod 9'15 mlcod 9'15 active append_log transaction_applied = 1 2022-03-29T17:49:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,687 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,687 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'16, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,687 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,688 [shard 0] osd - replicated_request(id=17, detail=RepRequest(from=1 req=osd_repop(client.4125.0:31 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'17, mlcod=9'17) v3)) start 2022-03-29T17:49:59.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,688 [shard 0] osd - replicated_request(id=18, detail=RepRequest(from=1 req=osd_repop(client.4125.0:32 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'18, mlcod=9'18) v3)) start 2022-03-29T17:49:59.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,689 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'16 (0'0,9'16] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'16 mlcod 9'16 active append_log log((0'0,9'16], crt=9'16) {9'17 (9'16) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:31 2022-03-29T17:49:59.681098+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,689 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'16 lcod 9'16 mlcod 9'16 active add_log_entry 9'17 (9'16) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:31 2022-03-29T17:49:59.681098+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,689 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'17 lcod 9'16 mlcod 9'16 active append_log approx pg log length = 17 2022-03-29T17:49:59.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,689 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'17 lcod 9'16 mlcod 9'16 active append_log transaction_applied = 1 2022-03-29T17:49:59.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,689 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,689 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'17, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,689 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,690 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'17 (0'0,9'17] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'17 mlcod 9'17 active append_log log((0'0,9'17], crt=9'17) {9'18 (9'17) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:32 2022-03-29T17:49:59.681132+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,690 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'17 lcod 9'17 mlcod 9'17 active add_log_entry 9'18 (9'17) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:32 2022-03-29T17:49:59.681132+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,690 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'18 lcod 9'17 mlcod 9'17 active append_log approx pg log length = 18 2022-03-29T17:49:59.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,690 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'18 lcod 9'17 mlcod 9'17 active append_log transaction_applied = 1 2022-03-29T17:49:59.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,690 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,690 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'18, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,691 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,691 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,691 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4125.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4125.0:34 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4125.0:34 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,692 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'18 lcod 9'15 mlcod 9'15 active+clean calc_trim_to_aggressive limit = 9'15 2022-03-29T17:49:59.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,693 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'18 lcod 9'15 mlcod 9'15 active+clean append_log log((0'0,9'18], crt=9'18) {9'19 (9'18) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:33 2022-03-29T17:49:59.681150+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,693 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'18 lcod 9'15 mlcod 9'15 active+clean add_log_entry 9'19 (9'18) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:33 2022-03-29T17:49:59.681150+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,693 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'19 lcod 9'15 mlcod 9'15 active+clean append_log approx pg log length = 19 2022-03-29T17:49:59.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,693 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'19 lcod 9'15 mlcod 9'15 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,693 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,693 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'19, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,693 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,694 [shard 0] osd - replicated_request(id=19, detail=RepRequest(from=1 req=osd_repop(client.4125.0:33 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'19, mlcod=9'19) v3)) start 2022-03-29T17:49:59.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,694 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'18 (0'0,9'18] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'18 mlcod 9'18 active append_log log((0'0,9'18], crt=9'18) {9'19 (9'18) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:33 2022-03-29T17:49:59.681150+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,694 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'18 lcod 9'18 mlcod 9'18 active add_log_entry 9'19 (9'18) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:33 2022-03-29T17:49:59.681150+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,694 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'19 lcod 9'18 mlcod 9'18 active append_log approx pg log length = 19 2022-03-29T17:49:59.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,694 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'19 lcod 9'18 mlcod 9'18 active append_log transaction_applied = 1 2022-03-29T17:49:59.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,694 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,694 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'19, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,694 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.738 INFO:tasks.ceph.osd.0.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/include/inline_memory.h:112:9: runtime error: load of misaligned address 0x62d0001228ef for type 'uint32_t', which requires 4 byte alignment 2022-03-29T17:49:59.739 INFO:tasks.ceph.osd.0.smithi138.stderr:0x62d0001228ef: note: pointer points here 2022-03-29T17:49:59.739 INFO:tasks.ceph.osd.0.smithi138.stderr: 04 00 00 00 0b 96 a4 68 02 00 00 00 1f 00 00 00 30 30 30 30 30 30 30 30 30 39 2e 30 30 30 30 30 2022-03-29T17:49:59.739 INFO:tasks.ceph.osd.0.smithi138.stderr: ^ 2022-03-29T17:49:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,695 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4125.0:34 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,695 [shard 0] osd - OpSequencer::start_op: op=34, last_started=33, last_unblocked=33, last_completed=29 2022-03-29T17:49:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,695 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4125.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4125.0:35 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4125.0:35 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'19 lcod 9'15 mlcod 9'15 active+clean calc_trim_to_aggressive limit = 9'15 2022-03-29T17:49:59.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'19 lcod 9'15 mlcod 9'15 active+clean append_log log((0'0,9'19], crt=9'19) {9'20 (9'19) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:34 2022-03-29T17:49:59.681170+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'19 lcod 9'15 mlcod 9'15 active+clean add_log_entry 9'20 (9'19) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:34 2022-03-29T17:49:59.681170+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,696 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'20 lcod 9'15 mlcod 9'15 active+clean append_log approx pg log length = 20 2022-03-29T17:49:59.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,697 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'15 lua=0'0 crt=9'20 lcod 9'15 mlcod 9'15 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,697 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,697 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'20, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,697 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,698 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4125.0:35 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,698 [shard 0] osd - OpSequencer::start_op: op=35, last_started=34, last_unblocked=34, last_completed=29 2022-03-29T17:49:59.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,698 [shard 0] osd - replicated_request(id=20, detail=RepRequest(from=1 req=osd_repop(client.4125.0:34 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'20, mlcod=9'20) v3)) start 2022-03-29T17:49:59.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,698 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'19 (0'0,9'19] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'19 mlcod 9'19 active append_log log((0'0,9'19], crt=9'19) {9'20 (9'19) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:34 2022-03-29T17:49:59.681170+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,698 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'19 lcod 9'19 mlcod 9'19 active add_log_entry 9'20 (9'19) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:34 2022-03-29T17:49:59.681170+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,698 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'20 lcod 9'19 mlcod 9'19 active append_log approx pg log length = 20 2022-03-29T17:49:59.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,698 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'20 lcod 9'19 mlcod 9'19 active append_log transaction_applied = 1 2022-03-29T17:49:59.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,699 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,699 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'20, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,699 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,699 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,699 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4125.0:36 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,699 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4125.0:36 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,700 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,700 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4125.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,700 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,700 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,700 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4125.0:36 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,700 [shard 0] osd - OpSequencer::start_op: op=36, last_started=35, last_unblocked=35, last_completed=29 2022-03-29T17:49:59.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'16 lua=0'0 crt=9'20 lcod 9'16 mlcod 9'16 active+clean calc_trim_to_aggressive limit = 9'16 2022-03-29T17:49:59.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'16 lua=0'0 crt=9'20 lcod 9'16 mlcod 9'16 active+clean append_log log((0'0,9'20], crt=9'20) {9'21 (9'20) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:35 2022-03-29T17:49:59.681191+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'16 lua=0'0 crt=9'20 lcod 9'16 mlcod 9'16 active+clean add_log_entry 9'21 (9'20) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:35 2022-03-29T17:49:59.681191+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'16 lua=0'0 crt=9'21 lcod 9'16 mlcod 9'16 active+clean append_log approx pg log length = 21 2022-03-29T17:49:59.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'16 lua=0'0 crt=9'21 lcod 9'16 mlcod 9'16 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,701 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,702 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'21, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,702 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,702 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,703 [shard 0] osd - replicated_request(id=21, detail=RepRequest(from=1 req=osd_repop(client.4125.0:35 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'21, mlcod=9'21) v3)) start 2022-03-29T17:49:59.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,704 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'20 (0'0,9'20] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'20 mlcod 9'20 active append_log log((0'0,9'20], crt=9'20) {9'21 (9'20) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:35 2022-03-29T17:49:59.681191+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,704 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'20 lcod 9'20 mlcod 9'20 active add_log_entry 9'21 (9'20) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:35 2022-03-29T17:49:59.681191+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,704 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'21 lcod 9'20 mlcod 9'20 active append_log approx pg log length = 21 2022-03-29T17:49:59.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,704 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'21 lcod 9'20 mlcod 9'20 active append_log transaction_applied = 1 2022-03-29T17:49:59.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,704 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,704 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'21, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,704 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.760 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:59,708 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@61720 >> mgr.4102 v2:172.21.15.138:6812/35195] closing: reset no, replace no 2022-03-29T17:49:59.761 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:49:59,708 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@61720 >> mgr.4102 v2:172.21.15.138:6812/35195] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:59.761 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:59,711 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@58956 >> mgr.4102 v2:172.21.15.138:6812/35195] closing: reset no, replace no 2022-03-29T17:49:59.762 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:49:59,712 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@58956 >> mgr.4102 v2:172.21.15.138:6812/35195] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:59.764 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,713 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,713 [shard 0] osd - OpSequencer::finish_op_in_order: op=30, last_started=36, last_unblocked=36, last_completed=29 2022-03-29T17:49:59.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,713 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4125.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,713 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4125.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - OpSequencer::finish_op_in_order: op=31, last_started=36, last_unblocked=36, last_completed=30 2022-03-29T17:49:59.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4125.0:37 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4125.0:37 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4125.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,714 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,715 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'17 lua=0'0 crt=9'21 lcod 9'17 mlcod 9'17 active+clean calc_trim_to_aggressive limit = 9'17 2022-03-29T17:49:59.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,715 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'17 lua=0'0 crt=9'21 lcod 9'17 mlcod 9'17 active+clean append_log log((0'0,9'21], crt=9'21) {9'22 (9'21) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:36 2022-03-29T17:49:59.681206+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,715 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'17 lua=0'0 crt=9'21 lcod 9'17 mlcod 9'17 active+clean add_log_entry 9'22 (9'21) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:36 2022-03-29T17:49:59.681206+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,715 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'17 lua=0'0 crt=9'22 lcod 9'17 mlcod 9'17 active+clean append_log approx pg log length = 22 2022-03-29T17:49:59.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,715 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'17 lua=0'0 crt=9'22 lcod 9'17 mlcod 9'17 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,715 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,715 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'22, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,715 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,716 [shard 0] osd - replicated_request(id=22, detail=RepRequest(from=1 req=osd_repop(client.4125.0:36 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'22, mlcod=9'22) v3)) start 2022-03-29T17:49:59.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,716 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'21 (0'0,9'21] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'21 mlcod 9'21 active append_log log((0'0,9'21], crt=9'21) {9'22 (9'21) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:36 2022-03-29T17:49:59.681206+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,716 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'21 lcod 9'21 mlcod 9'21 active add_log_entry 9'22 (9'21) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:36 2022-03-29T17:49:59.681206+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,716 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'22 lcod 9'21 mlcod 9'21 active append_log approx pg log length = 22 2022-03-29T17:49:59.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,716 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'22 lcod 9'21 mlcod 9'21 active append_log transaction_applied = 1 2022-03-29T17:49:59.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,716 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,716 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'22, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,716 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,717 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4125.0:37 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,717 [shard 0] osd - OpSequencer::start_op: op=37, last_started=36, last_unblocked=36, last_completed=31 2022-03-29T17:49:59.777 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,717 [shard 0] osd - handle_osd_map osd_map(9..10 src has 1..10) v4 2022-03-29T17:49:59.777 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,717 [shard 0] osd - handle_osd_map epochs [9..10], i have 9, src has [1..10] 2022-03-29T17:49:59.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,719 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:49:59.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,719 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - OpSequencer::finish_op_in_order: op=32, last_started=37, last_unblocked=37, last_completed=31 2022-03-29T17:49:59.779 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,720 [shard 0] osd - osd.1: committed_osd_maps(10, 10) 2022-03-29T17:49:59.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4125.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4125.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.780 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,720 [shard 0] osd - osd.1: map e 10 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:49:59.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.780 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,720 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.781 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,720 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.size 2022-03-29T17:49:59.781 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.781 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.781 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,721 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.version 2022-03-29T17:49:59.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'22 lcod 9'18 mlcod 9'18 active+clean calc_trim_to_aggressive limit = 9'18 2022-03-29T17:49:59.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'22 lcod 9'18 mlcod 9'18 active+clean append_log log((0'0,9'22], crt=9'22) {9'23 (9'22) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:37 2022-03-29T17:49:59.681234+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'22 lcod 9'18 mlcod 9'18 active+clean add_log_entry 9'23 (9'22) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:37 2022-03-29T17:49:59.681234+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean append_log approx pg log length = 23 2022-03-29T17:49:59.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'23, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,721 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.784 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,724 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@54668 >> mgr.4102 v2:172.21.15.138:6812/35195] closing: reset no, replace no 2022-03-29T17:49:59.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,725 [shard 0] osd - pg_advance_map(id=3, detail=PGAdvanceMap(pg=1.0 from=9 to=10)): start 2022-03-29T17:49:59.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,725 [shard 0] osd - replicated_request(id=23, detail=RepRequest(from=1 req=osd_repop(client.4125.0:37 1.0 e9/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 9'23, mlcod=9'23) v3)) start 2022-03-29T17:49:59.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,725 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'22 (0'0,9'22] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'22 mlcod 9'22 active append_log log((0'0,9'22], crt=9'22) {9'23 (9'22) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:37 2022-03-29T17:49:59.681234+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,726 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'22 lcod 9'22 mlcod 9'22 active add_log_entry 9'23 (9'22) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:37 2022-03-29T17:49:59.681234+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,726 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'23 lcod 9'22 mlcod 9'22 active append_log approx pg log length = 23 2022-03-29T17:49:59.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,726 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'23 lcod 9'22 mlcod 9'22 active append_log transaction_applied = 1 2022-03-29T17:49:59.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,726 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,726 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 9'23, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,726 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.793 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,726 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:49:59.794 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,726 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean state: Active advmap 2022-03-29T17:49:59.794 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean state: Started advmap 2022-03-29T17:49:59.794 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:49:59.794 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean activate_map 2022-03-29T17:49:59.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean state: Active: handling ActMap 2022-03-29T17:49:59.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean state: handle ActMap primary 2022-03-29T17:49:59.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:49:59.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean publish_stats_to_osd 9:9 2022-03-29T17:49:59.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'18 lua=0'0 crt=9'23 lcod 9'18 mlcod 9'18 active+clean activate_map: Not dirtying info: last_persisted is 9 while current is 10 2022-03-29T17:49:59.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - log is not dirty 2022-03-29T17:49:59.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,727 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:49:59.796 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,727 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@54668 >> mgr.4102 v2:172.21.15.138:6812/35195] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:49:59.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,728 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,728 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,728 [shard 0] osd - pg_advance_map(id=3, detail=PGAdvanceMap(pg=1.0 from=9 to=10)): complete 2022-03-29T17:49:59.797 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,728 [shard 0] osd - osd.1: now active 2022-03-29T17:49:59.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,729 [shard 0] osd - OpSequencer::finish_op_in_order: op=33, last_started=37, last_unblocked=37, last_completed=32 2022-03-29T17:49:59.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,729 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4125.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,729 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,729 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,729 [shard 0] osd - OpSequencer::finish_op_in_order: op=34, last_started=37, last_unblocked=37, last_completed=33 2022-03-29T17:49:59.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,730 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4125.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,730 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,730 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,730 [shard 0] osd - OpSequencer::finish_op_in_order: op=35, last_started=37, last_unblocked=37, last_completed=34 2022-03-29T17:49:59.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,730 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4125.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,731 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,731 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,731 [shard 0] osd - OpSequencer::finish_op_in_order: op=36, last_started=37, last_unblocked=37, last_completed=35 2022-03-29T17:49:59.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,731 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4125.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,731 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,731 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,731 [shard 0] osd - OpSequencer::finish_op_in_order: op=37, last_started=37, last_unblocked=37, last_completed=36 2022-03-29T17:49:59.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,731 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4125.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,743 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4125.0:38 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,743 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4125.0:38 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4125.0:38 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - OpSequencer::start_op: op=38, last_started=37, last_unblocked=37, last_completed=37 2022-03-29T17:49:59.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4125.0:38 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4125.0:39 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4125.0:39 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:49:59.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'23 mlcod 9'23 active+clean calc_trim_to_aggressive limit = 9'23 2022-03-29T17:49:59.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - pg_epoch 10 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=9'23 mlcod 9'23 active+clean append_log log((0'0,9'23], crt=9'23) {10'24 (9'15) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:38 2022-03-29T17:49:59.743433+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'23 lua=0'0 crt=9'23 lcod 9'23 mlcod 9'23 active+clean add_log_entry 10'24 (9'15) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:38 2022-03-29T17:49:59.743433+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'23 lua=0'0 crt=10'24 lcod 9'23 mlcod 9'23 active+clean append_log approx pg log length = 24 2022-03-29T17:49:59.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'23 lua=0'0 crt=10'24 lcod 9'23 mlcod 9'23 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,744 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'24, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,745 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,746 [shard 0] osd - replicated_request(id=24, detail=RepRequest(from=1 req=osd_repop(client.4125.0:38 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'24, mlcod=10'24) v3)) start 2022-03-29T17:49:59.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,746 [shard 0] osd - pg_epoch 9 pg[1.0( v 9'23 (0'0,9'23] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'23 mlcod 9'23 active append_log log((0'0,9'23], crt=9'23) {10'24 (9'15) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:38 2022-03-29T17:49:59.743433+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,746 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=9'23 lcod 9'23 mlcod 9'23 active add_log_entry 10'24 (9'15) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:38 2022-03-29T17:49:59.743433+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,746 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'24 lcod 9'23 mlcod 9'23 active append_log approx pg log length = 24 2022-03-29T17:49:59.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,746 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'24 lcod 9'23 mlcod 9'23 active append_log transaction_applied = 1 2022-03-29T17:49:59.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,746 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,746 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'24, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,746 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,747 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4125.0:39 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,747 [shard 0] osd - OpSequencer::start_op: op=39, last_started=38, last_unblocked=38, last_completed=37 2022-03-29T17:49:59.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4125.0:39 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:49:59.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,748 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'23 lua=0'0 crt=10'24 lcod 9'23 mlcod 9'23 active+clean calc_trim_to_aggressive limit = 9'23 2022-03-29T17:49:59.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,749 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'23 lua=0'0 crt=10'24 lcod 9'23 mlcod 9'23 active+clean append_log log((0'0,10'24], crt=10'24) {10'25 (10'24) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:39 2022-03-29T17:49:59.743483+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,749 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'23 lua=0'0 crt=10'24 lcod 9'23 mlcod 9'23 active+clean add_log_entry 10'25 (10'24) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:39 2022-03-29T17:49:59.743483+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,749 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'23 lua=0'0 crt=10'25 lcod 9'23 mlcod 9'23 active+clean append_log approx pg log length = 25 2022-03-29T17:49:59.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,749 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=9'23 lua=0'0 crt=10'25 lcod 9'23 mlcod 9'23 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,749 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,749 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'25, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,749 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,750 [shard 0] osd - replicated_request(id=25, detail=RepRequest(from=1 req=osd_repop(client.4125.0:39 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'25, mlcod=10'25) v3)) start 2022-03-29T17:49:59.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,750 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'24 (0'0,10'24] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'24 mlcod 10'24 active append_log log((0'0,10'24], crt=10'24) {10'25 (10'24) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:39 2022-03-29T17:49:59.743483+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,750 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'24 lcod 10'24 mlcod 10'24 active add_log_entry 10'25 (10'24) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:39 2022-03-29T17:49:59.743483+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,750 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'25 lcod 10'24 mlcod 10'24 active append_log approx pg log length = 25 2022-03-29T17:49:59.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,750 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'25 lcod 10'24 mlcod 10'24 active append_log transaction_applied = 1 2022-03-29T17:49:59.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,750 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,750 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'25, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,750 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,751 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:38 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,751 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,751 [shard 0] osd - OpSequencer::finish_op_in_order: op=38, last_started=39, last_unblocked=39, last_completed=37 2022-03-29T17:49:59.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,752 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4125.0:38 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,753 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:39 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,753 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,753 [shard 0] osd - OpSequencer::finish_op_in_order: op=39, last_started=39, last_unblocked=39, last_completed=38 2022-03-29T17:49:59.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,754 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4125.0:39 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,792 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4125.0:40 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,792 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4125.0:40 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,792 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4125.0:40 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,792 [shard 0] osd - OpSequencer::start_op: op=40, last_started=39, last_unblocked=39, last_completed=39 2022-03-29T17:49:59.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4125.0:40 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:49:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:49:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:49:59.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:49:59.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'25 mlcod 10'25 active+clean calc_trim_to_aggressive limit = 10'25 2022-03-29T17:49:59.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'25 mlcod 10'25 active+clean append_log log((0'0,10'25], crt=10'25) {10'26 (9'23) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:40 2022-03-29T17:49:59.792402+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'25 lua=0'0 crt=10'25 lcod 10'25 mlcod 10'25 active+clean add_log_entry 10'26 (9'23) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:40 2022-03-29T17:49:59.792402+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'25 lua=0'0 crt=10'26 lcod 10'25 mlcod 10'25 active+clean append_log approx pg log length = 26 2022-03-29T17:49:59.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'25 lua=0'0 crt=10'26 lcod 10'25 mlcod 10'25 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'26, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,793 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,794 [shard 0] osd - replicated_request(id=26, detail=RepRequest(from=1 req=osd_repop(client.4125.0:40 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'26, mlcod=10'26) v3)) start 2022-03-29T17:49:59.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,795 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'25 (0'0,10'25] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'25 mlcod 10'25 active append_log log((0'0,10'25], crt=10'25) {10'26 (9'23) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:40 2022-03-29T17:49:59.792402+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,795 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'25 lcod 10'25 mlcod 10'25 active add_log_entry 10'26 (9'23) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:40 2022-03-29T17:49:59.792402+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,795 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'26 lcod 10'25 mlcod 10'25 active append_log approx pg log length = 26 2022-03-29T17:49:59.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,795 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'26 lcod 10'25 mlcod 10'25 active append_log transaction_applied = 1 2022-03-29T17:49:59.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,795 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,795 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'26, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,795 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,835 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:40 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:49:59.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:49:59.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - OpSequencer::finish_op_in_order: op=40, last_started=40, last_unblocked=40, last_completed=39 2022-03-29T17:49:59.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4125.0:40 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4125.0:41 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4125.0:41 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4125.0:41 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - OpSequencer::start_op: op=41, last_started=40, last_unblocked=40, last_completed=40 2022-03-29T17:49:59.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4125.0:41 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,836 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:49:59.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:49:59.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.845 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,837 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:49:59.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:49:59.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,837 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.847 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,838 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:49:59.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'26 mlcod 10'26 active+clean calc_trim_to_aggressive limit = 10'26 2022-03-29T17:49:59.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'26 mlcod 10'26 active+clean append_log log((0'0,10'26], crt=10'26) {10'27 (10'25) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:41 2022-03-29T17:49:59.836378+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'26 lua=0'0 crt=10'26 lcod 10'26 mlcod 10'26 active+clean add_log_entry 10'27 (10'25) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:41 2022-03-29T17:49:59.836378+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'26 lua=0'0 crt=10'27 lcod 10'26 mlcod 10'26 active+clean append_log approx pg log length = 27 2022-03-29T17:49:59.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'26 lua=0'0 crt=10'27 lcod 10'26 mlcod 10'26 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'27, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,838 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,876 [shard 0] osd - replicated_request(id=27, detail=RepRequest(from=1 req=osd_repop(client.4125.0:41 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'27, mlcod=10'27) v3)) start 2022-03-29T17:49:59.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,876 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'26 (0'0,10'26] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'26 mlcod 10'26 active append_log log((0'0,10'26], crt=10'26) {10'27 (10'25) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:41 2022-03-29T17:49:59.836378+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,876 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'26 lcod 10'26 mlcod 10'26 active add_log_entry 10'27 (10'25) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:41 2022-03-29T17:49:59.836378+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,876 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'27 lcod 10'26 mlcod 10'26 active append_log approx pg log length = 27 2022-03-29T17:49:59.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,876 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'27 lcod 10'26 mlcod 10'26 active append_log transaction_applied = 1 2022-03-29T17:49:59.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,876 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,877 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'27, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,877 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:49:59.903 INFO:teuthology.misc.health.smithi138.stdout: 2022-03-29T17:49:59.903 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"3be68405-f255-4011-9a3b-ba646329d493","created":"2022-03-29T17:49:51.193768+0000","modified":"2022-03-29T17:49:59.693795+0000","last_up_change":"2022-03-29T17:49:55.205833+0000","last_in_change":"2022-03-29T17:49:52.178337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-29T17:49:56.727900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3e4be414-1d2c-4bc5-b6b2-d1f7948d012a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":35577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":35577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":35577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":35577}]},"public_addr":"172.21.15.138:6805/35577","cluster_addr":"172.21.15.138:6804/35577","heartbeat_back_addr":"172.21.15.138:6806/35577","heartbeat_front_addr":"172.21.15.138:6809/35577","state":["exists","up"]},{"osd":1,"uuid":"59cc9d2e-e40e-4048-b5cb-31db1b464c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":35578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":35578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":35578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":35578}]},"public_addr":"172.21.15.138:6800/35578","cluster_addr":"172.21.15.138:6801/35578","heartbeat_back_addr":"172.21.15.138:6807/35578","heartbeat_front_addr":"172.21.15.138:6808/35578","state":["exists","up"]},{"osd":2,"uuid":"88a5bd02-6c7b-4069-9fdb-aea1c201b934","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":35579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":35579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":35579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":35579}]},"public_addr":"172.21.15.138:6803/35579","cluster_addr":"172.21.15.138:6802/35579","heartbeat_back_addr":"172.21.15.138:6811/35579","heartbeat_front_addr":"172.21.15.138:6810/35579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-29T17:49:59.916 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-03-29T17:49:59.916 INFO:tasks.ceph:Creating RBD pool 2022-03-29T17:49:59.917 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd pool create rbd 8 2022-03-29T17:49:59.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,918 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:41 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:49:59.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,919 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,919 [shard 0] osd - OpSequencer::finish_op_in_order: op=41, last_started=41, last_unblocked=41, last_completed=40 2022-03-29T17:49:59.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,919 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4125.0:41 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:49:59.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,919 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4125.0:42 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:49:59.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,919 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4125.0:42 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:49:59.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4125.0:42 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:49:59.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - OpSequencer::start_op: op=42, last_started=41, last_unblocked=41, last_completed=41 2022-03-29T17:49:59.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4125.0:42 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:49:59.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:49:59.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:49:59.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:49:59.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,920 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,921 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:49:59.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:49:59.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:49:59.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:49:59.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:49:59.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:49:59.933 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:49:59,921 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:49:59.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:49:59.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:49:59.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:49:59.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'27 mlcod 10'27 active+clean calc_trim_to_aggressive limit = 10'27 2022-03-29T17:49:59.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'27 mlcod 10'27 active+clean append_log log((0'0,10'27], crt=10'27) {10'28 (10'27) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:42 2022-03-29T17:49:59.919286+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'27 lua=0'0 crt=10'27 lcod 10'27 mlcod 10'27 active+clean add_log_entry 10'28 (10'27) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:42 2022-03-29T17:49:59.919286+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'27 lua=0'0 crt=10'28 lcod 10'27 mlcod 10'27 active+clean append_log approx pg log length = 28 2022-03-29T17:49:59.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'27 lua=0'0 crt=10'28 lcod 10'27 mlcod 10'27 active+clean append_log transaction_applied = 1 2022-03-29T17:49:59.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,921 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'28, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:49:59,922 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:49:59.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,959 [shard 0] osd - replicated_request(id=28, detail=RepRequest(from=1 req=osd_repop(client.4125.0:42 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'28, mlcod=10'28) v3)) start 2022-03-29T17:49:59.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,960 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'27 (0'0,10'27] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'27 mlcod 10'27 active append_log log((0'0,10'27], crt=10'27) {10'28 (10'27) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:42 2022-03-29T17:49:59.919286+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:49:59.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,960 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'27 lcod 10'27 mlcod 10'27 active add_log_entry 10'28 (10'27) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:42 2022-03-29T17:49:59.919286+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:49:59.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,960 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'28 lcod 10'27 mlcod 10'27 active append_log approx pg log length = 28 2022-03-29T17:49:59.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,960 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'28 lcod 10'27 mlcod 10'27 active append_log transaction_applied = 1 2022-03-29T17:49:59.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,960 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:49:59.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,960 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'28, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:49:59.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:49:59,960 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,000 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:42 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,001 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,001 [shard 0] osd - OpSequencer::finish_op_in_order: op=42, last_started=42, last_unblocked=42, last_completed=41 2022-03-29T17:50:00.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,001 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4125.0:42 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,001 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4125.0:43 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,001 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4125.0:43 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,001 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4125.0:43 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,001 [shard 0] osd - OpSequencer::start_op: op=43, last_started=42, last_unblocked=42, last_completed=42 2022-03-29T17:50:00.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,002 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,002 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4125.0:43 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,002 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,002 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,002 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,002 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,002 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:00.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:50:00.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:00.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,003 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:43 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - OpSequencer::finish_op_in_order: op=43, last_started=43, last_unblocked=43, last_completed=42 2022-03-29T17:50:00.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4125.0:43 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4125.0:44 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4125.0:44 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4125.0:44 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - OpSequencer::start_op: op=44, last_started=43, last_unblocked=43, last_completed=43 2022-03-29T17:50:00.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4125.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,004 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op read 2022-03-29T17:50:00.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - handling op read on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - read: data length: 1 2022-03-29T17:50:00.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - OpSequencer::finish_op_in_order: op=44, last_started=44, last_unblocked=44, last_completed=43 2022-03-29T17:50:00.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,005 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4125.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4125.0:45 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4125.0:45 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4125.0:45 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - OpSequencer::start_op: op=45, last_started=44, last_unblocked=44, last_completed=44 2022-03-29T17:50:00.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4125.0:45 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:50:00.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,006 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,007 [shard 0] osd - read: data length: 16 2022-03-29T17:50:00.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,007 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,007 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:45 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,007 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,007 [shard 0] osd - OpSequencer::finish_op_in_order: op=45, last_started=45, last_unblocked=45, last_completed=44 2022-03-29T17:50:00.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,007 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4125.0:45 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,007 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4125.0:46 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,007 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4125.0:46 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4125.0:46 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - OpSequencer::start_op: op=46, last_started=45, last_unblocked=45, last_completed=45 2022-03-29T17:50:00.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4125.0:46 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:50:00.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:50:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - OpSequencer::finish_op_in_order: op=46, last_started=46, last_unblocked=46, last_completed=45 2022-03-29T17:50:00.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,008 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4125.0:46 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4125.0:47 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4125.0:47 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4125.0:47 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - OpSequencer::start_op: op=47, last_started=46, last_unblocked=46, last_completed=46 2022-03-29T17:50:00.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4125.0:47 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:00.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:50:00.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.037 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,009 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:00.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,009 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:50:00.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.040 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,010 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:50:00.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'28 mlcod 10'28 active+clean calc_trim_to_aggressive limit = 10'28 2022-03-29T17:50:00.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'28 mlcod 10'28 active+clean append_log log((0'0,10'28], crt=10'28) {10'29 (10'28) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:47 2022-03-29T17:50:00.009011+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'28 lua=0'0 crt=10'28 lcod 10'28 mlcod 10'28 active+clean add_log_entry 10'29 (10'28) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:47 2022-03-29T17:50:00.009011+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'28 lua=0'0 crt=10'29 lcod 10'28 mlcod 10'28 active+clean append_log approx pg log length = 29 2022-03-29T17:50:00.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'28 lua=0'0 crt=10'29 lcod 10'28 mlcod 10'28 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,010 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'29, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,011 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.044 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,041 [shard 0] osd - replicated_request(id=29, detail=RepRequest(from=1 req=osd_repop(client.4125.0:47 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'29, mlcod=10'29) v3)) start 2022-03-29T17:50:00.044 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,042 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'28 (0'0,10'28] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'28 mlcod 10'28 active append_log log((0'0,10'28], crt=10'28) {10'29 (10'28) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:47 2022-03-29T17:50:00.009011+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,042 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'28 lcod 10'28 mlcod 10'28 active add_log_entry 10'29 (10'28) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:47 2022-03-29T17:50:00.009011+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,042 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'29 lcod 10'28 mlcod 10'28 active append_log approx pg log length = 29 2022-03-29T17:50:00.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,042 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'29 lcod 10'28 mlcod 10'28 active append_log transaction_applied = 1 2022-03-29T17:50:00.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,042 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,042 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'29, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,042 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,082 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:47 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,083 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,083 [shard 0] osd - OpSequencer::finish_op_in_order: op=47, last_started=47, last_unblocked=47, last_completed=46 2022-03-29T17:50:00.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,083 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4125.0:47 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4125.0:48 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4125.0:48 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4125.0:48 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - OpSequencer::start_op: op=48, last_started=47, last_unblocked=47, last_completed=47 2022-03-29T17:50:00.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4125.0:48 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:00.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,084 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:50:00.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.097 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,085 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:50:00.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:00.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:50:00.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.099 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,085 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,085 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'29 mlcod 10'29 active+clean calc_trim_to_aggressive limit = 10'29 2022-03-29T17:50:00.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,086 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'29 mlcod 10'29 active+clean append_log log((0'0,10'29], crt=10'29) {10'30 (10'29) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:48 2022-03-29T17:50:00.083854+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,086 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'29 lua=0'0 crt=10'29 lcod 10'29 mlcod 10'29 active+clean add_log_entry 10'30 (10'29) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:48 2022-03-29T17:50:00.083854+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,086 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'29 lua=0'0 crt=10'30 lcod 10'29 mlcod 10'29 active+clean append_log approx pg log length = 30 2022-03-29T17:50:00.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,086 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'29 lua=0'0 crt=10'30 lcod 10'29 mlcod 10'29 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,086 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,086 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'30, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,086 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.124 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,123 [shard 0] osd - replicated_request(id=30, detail=RepRequest(from=1 req=osd_repop(client.4125.0:48 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'30, mlcod=10'30) v3)) start 2022-03-29T17:50:00.126 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,123 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'29 (0'0,10'29] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'29 mlcod 10'29 active append_log log((0'0,10'29], crt=10'29) {10'30 (10'29) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:48 2022-03-29T17:50:00.083854+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.126 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,124 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'29 lcod 10'29 mlcod 10'29 active add_log_entry 10'30 (10'29) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:48 2022-03-29T17:50:00.083854+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.127 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,124 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'30 lcod 10'29 mlcod 10'29 active append_log approx pg log length = 30 2022-03-29T17:50:00.127 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,124 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'30 lcod 10'29 mlcod 10'29 active append_log transaction_applied = 1 2022-03-29T17:50:00.127 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,124 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.127 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,124 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'30, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.128 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,124 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,164 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:48 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,165 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,165 [shard 0] osd - OpSequencer::finish_op_in_order: op=48, last_started=48, last_unblocked=48, last_completed=47 2022-03-29T17:50:00.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,165 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4125.0:48 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,165 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4125.0:49 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,165 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4125.0:49 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,165 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4125.0:49 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,165 [shard 0] osd - OpSequencer::start_op: op=49, last_started=48, last_unblocked=48, last_completed=48 2022-03-29T17:50:00.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,165 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4125.0:49 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:00.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:50:00.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:00.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:49 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,166 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - OpSequencer::finish_op_in_order: op=49, last_started=49, last_unblocked=49, last_completed=48 2022-03-29T17:50:00.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4125.0:49 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4125.0:50 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4125.0:50 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4125.0:50 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - OpSequencer::start_op: op=50, last_started=49, last_unblocked=49, last_completed=49 2022-03-29T17:50:00.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4125.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op read 2022-03-29T17:50:00.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - handling op read on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - read: data length: 1 2022-03-29T17:50:00.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,167 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - OpSequencer::finish_op_in_order: op=50, last_started=50, last_unblocked=50, last_completed=49 2022-03-29T17:50:00.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4125.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4125.0:51 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4125.0:51 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4125.0:51 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - OpSequencer::start_op: op=51, last_started=50, last_unblocked=50, last_completed=50 2022-03-29T17:50:00.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4125.0:51 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:50:00.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - read: data length: 16 2022-03-29T17:50:00.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,168 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:51 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - OpSequencer::finish_op_in_order: op=51, last_started=51, last_unblocked=51, last_completed=50 2022-03-29T17:50:00.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4125.0:51 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4125.0:52 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4125.0:52 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4125.0:52 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - OpSequencer::start_op: op=52, last_started=51, last_unblocked=51, last_completed=51 2022-03-29T17:50:00.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4125.0:52 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:50:00.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,169 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:50:00.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - OpSequencer::finish_op_in_order: op=52, last_started=52, last_unblocked=52, last_completed=51 2022-03-29T17:50:00.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4125.0:52 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4125.0:53 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4125.0:53 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4125.0:53 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - OpSequencer::start_op: op=53, last_started=52, last_unblocked=52, last_completed=52 2022-03-29T17:50:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4125.0:53 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4125.0:54 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,170 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4125.0:54 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'30 mlcod 10'30 active+clean calc_trim_to_aggressive limit = 10'30 2022-03-29T17:50:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'30 mlcod 10'30 active+clean append_log log((0'0,10'30], crt=10'30) {10'31 (10'26) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:53 2022-03-29T17:50:00.170204+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'30 lcod 10'30 mlcod 10'30 active+clean add_log_entry 10'31 (10'26) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:53 2022-03-29T17:50:00.170204+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'31 lcod 10'30 mlcod 10'30 active+clean append_log approx pg log length = 31 2022-03-29T17:50:00.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'31 lcod 10'30 mlcod 10'30 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'31, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,171 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4125.0:54 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - OpSequencer::start_op: op=54, last_started=53, last_unblocked=53, last_completed=52 2022-03-29T17:50:00.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4125.0:54 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'31 lcod 10'30 mlcod 10'30 active+clean calc_trim_to_aggressive limit = 10'30 2022-03-29T17:50:00.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'31 lcod 10'30 mlcod 10'30 active+clean append_log log((0'0,10'31], crt=10'31) {10'32 (10'31) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:54 2022-03-29T17:50:00.170245+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,172 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'31 lcod 10'30 mlcod 10'30 active+clean add_log_entry 10'32 (10'31) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:54 2022-03-29T17:50:00.170245+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'32 lcod 10'30 mlcod 10'30 active+clean append_log approx pg log length = 32 2022-03-29T17:50:00.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'32 lcod 10'30 mlcod 10'30 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'32, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4125.0:55 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4125.0:55 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4125.0:55 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,173 [shard 0] osd - OpSequencer::start_op: op=55, last_started=54, last_unblocked=54, last_completed=52 2022-03-29T17:50:00.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4125.0:55 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'32 lcod 10'30 mlcod 10'30 active+clean calc_trim_to_aggressive limit = 10'30 2022-03-29T17:50:00.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'32 lcod 10'30 mlcod 10'30 active+clean append_log log((0'0,10'32], crt=10'32) {10'33 (10'32) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:55 2022-03-29T17:50:00.170280+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'32 lcod 10'30 mlcod 10'30 active+clean add_log_entry 10'33 (10'32) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:55 2022-03-29T17:50:00.170280+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'33 lcod 10'30 mlcod 10'30 active+clean append_log approx pg log length = 33 2022-03-29T17:50:00.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'33 lcod 10'30 mlcod 10'30 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'33, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4125.0:56 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,174 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4125.0:56 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,175 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4125.0:56 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,175 [shard 0] osd - OpSequencer::start_op: op=56, last_started=55, last_unblocked=55, last_completed=52 2022-03-29T17:50:00.214 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,175 [shard 0] osd - replicated_request(id=31, detail=RepRequest(from=1 req=osd_repop(client.4125.0:53 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'31, mlcod=10'31) v3)) start 2022-03-29T17:50:00.214 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,176 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'30 (0'0,10'30] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'30 mlcod 10'30 active append_log log((0'0,10'30], crt=10'30) {10'31 (10'26) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:53 2022-03-29T17:50:00.170204+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.215 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,176 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'30 lcod 10'30 mlcod 10'30 active add_log_entry 10'31 (10'26) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:53 2022-03-29T17:50:00.170204+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.215 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,176 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'31 lcod 10'30 mlcod 10'30 active append_log approx pg log length = 31 2022-03-29T17:50:00.215 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,176 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'31 lcod 10'30 mlcod 10'30 active append_log transaction_applied = 1 2022-03-29T17:50:00.215 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,176 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.216 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,176 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'31, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.216 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,176 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.216 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,176 [shard 0] osd - replicated_request(id=32, detail=RepRequest(from=1 req=osd_repop(client.4125.0:54 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'32, mlcod=10'32) v3)) start 2022-03-29T17:50:00.216 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,177 [shard 0] osd - replicated_request(id=33, detail=RepRequest(from=1 req=osd_repop(client.4125.0:55 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'33, mlcod=10'33) v3)) start 2022-03-29T17:50:00.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,177 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'31 (0'0,10'31] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'31 mlcod 10'31 active append_log log((0'0,10'31], crt=10'31) {10'32 (10'31) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:54 2022-03-29T17:50:00.170245+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,177 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'31 lcod 10'31 mlcod 10'31 active add_log_entry 10'32 (10'31) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:54 2022-03-29T17:50:00.170245+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,177 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'32 lcod 10'31 mlcod 10'31 active append_log approx pg log length = 32 2022-03-29T17:50:00.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,177 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'32 lcod 10'31 mlcod 10'31 active append_log transaction_applied = 1 2022-03-29T17:50:00.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,177 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,178 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'32, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,178 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'32 (0'0,10'32] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'32 mlcod 10'32 active append_log log((0'0,10'32], crt=10'32) {10'33 (10'32) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:55 2022-03-29T17:50:00.170280+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'32 lcod 10'32 mlcod 10'32 active add_log_entry 10'33 (10'32) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:55 2022-03-29T17:50:00.170280+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.219 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'33 lcod 10'32 mlcod 10'32 active append_log approx pg log length = 33 2022-03-29T17:50:00.219 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'33 lcod 10'32 mlcod 10'32 active append_log transaction_applied = 1 2022-03-29T17:50:00.219 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.219 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'33, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4125.0:56 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,179 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'33 lcod 10'30 mlcod 10'30 active+clean calc_trim_to_aggressive limit = 10'30 2022-03-29T17:50:00.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'33 lcod 10'30 mlcod 10'30 active+clean append_log log((0'0,10'33], crt=10'33) {10'34 (10'33) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:56 2022-03-29T17:50:00.170304+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'33 lcod 10'30 mlcod 10'30 active+clean add_log_entry 10'34 (10'33) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:56 2022-03-29T17:50:00.170304+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'34 lcod 10'30 mlcod 10'30 active+clean append_log approx pg log length = 34 2022-03-29T17:50:00.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'30 lua=0'0 crt=10'34 lcod 10'30 mlcod 10'30 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,180 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,181 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'34, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,181 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.226 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,182 [shard 0] osd - replicated_request(id=34, detail=RepRequest(from=1 req=osd_repop(client.4125.0:56 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'34, mlcod=10'34) v3)) start 2022-03-29T17:50:00.226 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,182 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'33 (0'0,10'33] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'33 mlcod 10'33 active append_log log((0'0,10'33], crt=10'33) {10'34 (10'33) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:56 2022-03-29T17:50:00.170304+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,182 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'33 lcod 10'33 mlcod 10'33 active add_log_entry 10'34 (10'33) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:56 2022-03-29T17:50:00.170304+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,182 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'34 lcod 10'33 mlcod 10'33 active append_log approx pg log length = 34 2022-03-29T17:50:00.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,182 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'34 lcod 10'33 mlcod 10'33 active append_log transaction_applied = 1 2022-03-29T17:50:00.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,182 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.228 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,182 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'34, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.228 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,182 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,183 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:53 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,184 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,184 [shard 0] osd - OpSequencer::finish_op_in_order: op=53, last_started=56, last_unblocked=56, last_completed=52 2022-03-29T17:50:00.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,184 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4125.0:53 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,184 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:54 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,185 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,185 [shard 0] osd - OpSequencer::finish_op_in_order: op=54, last_started=56, last_unblocked=56, last_completed=53 2022-03-29T17:50:00.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,185 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4125.0:54 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,185 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:55 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,186 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,186 [shard 0] osd - OpSequencer::finish_op_in_order: op=55, last_started=56, last_unblocked=56, last_completed=54 2022-03-29T17:50:00.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,186 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4125.0:55 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,186 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:56 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,186 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,186 [shard 0] osd - OpSequencer::finish_op_in_order: op=56, last_started=56, last_unblocked=56, last_completed=55 2022-03-29T17:50:00.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,186 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4125.0:56 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,226 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4125.0:57 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,226 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4125.0:57 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,226 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4125.0:57 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,226 [shard 0] osd - OpSequencer::start_op: op=57, last_started=56, last_unblocked=56, last_completed=56 2022-03-29T17:50:00.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,226 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,226 [shard 0] osd - client_request(id=58, detail=m=[osd_op(client.4125.0:58 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,226 [shard 0] osd - client_request(id=58, detail=m=[osd_op(client.4125.0:58 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,226 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4125.0:57 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,227 [shard 0] alienstore - stat 2022-03-29T17:50:00.237 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,227 [shard 0] alienstore - stat 2022-03-29T17:50:00.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,229 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.239 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,229 [shard 0] alienstore - stat 2022-03-29T17:50:00.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,229 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,229 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:00.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,229 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,229 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,229 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,230 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'34 mlcod 10'34 active+clean calc_trim_to_aggressive limit = 10'34 2022-03-29T17:50:00.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,230 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'34 mlcod 10'34 active+clean append_log log((0'0,10'34], crt=10'34) {10'35 (10'30) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:57 2022-03-29T17:50:00.225408+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,230 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'34 lua=0'0 crt=10'34 lcod 10'34 mlcod 10'34 active+clean add_log_entry 10'35 (10'30) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:57 2022-03-29T17:50:00.225408+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,230 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'34 lua=0'0 crt=10'35 lcod 10'34 mlcod 10'34 active+clean append_log approx pg log length = 35 2022-03-29T17:50:00.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,230 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'34 lua=0'0 crt=10'35 lcod 10'34 mlcod 10'34 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,230 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,230 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'35, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,230 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.243 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,231 [shard 0] osd - replicated_request(id=35, detail=RepRequest(from=1 req=osd_repop(client.4125.0:57 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'35, mlcod=10'35) v3)) start 2022-03-29T17:50:00.243 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,231 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'34 (0'0,10'34] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'34 mlcod 10'34 active append_log log((0'0,10'34], crt=10'34) {10'35 (10'30) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:57 2022-03-29T17:50:00.225408+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.244 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,231 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'34 lcod 10'34 mlcod 10'34 active add_log_entry 10'35 (10'30) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:57 2022-03-29T17:50:00.225408+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.244 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,231 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'35 lcod 10'34 mlcod 10'34 active append_log approx pg log length = 35 2022-03-29T17:50:00.244 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,231 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'35 lcod 10'34 mlcod 10'34 active append_log transaction_applied = 1 2022-03-29T17:50:00.244 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,231 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.244 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,231 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'35, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.245 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,232 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,232 [shard 0] osd - client_request(id=58, detail=m=[osd_op(client.4125.0:58 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,232 [shard 0] osd - OpSequencer::start_op: op=58, last_started=57, last_unblocked=57, last_completed=56 2022-03-29T17:50:00.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,233 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,234 [shard 0] osd - client_request(id=58, detail=m=[osd_op(client.4125.0:58 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 131072~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,234 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,234 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,234 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:00.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,234 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,234 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'34 lua=0'0 crt=10'35 lcod 10'34 mlcod 10'34 active+clean calc_trim_to_aggressive limit = 10'34 2022-03-29T17:50:00.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'34 lua=0'0 crt=10'35 lcod 10'34 mlcod 10'34 active+clean append_log log((0'0,10'35], crt=10'35) {10'36 (10'35) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:58 2022-03-29T17:50:00.225471+0000 0 ObjectCleanRegions clean_offsets: [0~131072,196608~18446744073709355007], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'34 lua=0'0 crt=10'35 lcod 10'34 mlcod 10'34 active+clean add_log_entry 10'36 (10'35) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:58 2022-03-29T17:50:00.225471+0000 0 ObjectCleanRegions clean_offsets: [0~131072,196608~18446744073709355007], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'34 lua=0'0 crt=10'36 lcod 10'34 mlcod 10'34 active+clean append_log approx pg log length = 36 2022-03-29T17:50:00.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'34 lua=0'0 crt=10'36 lcod 10'34 mlcod 10'34 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'36, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,235 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.252 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,236 [shard 0] osd - replicated_request(id=36, detail=RepRequest(from=1 req=osd_repop(client.4125.0:58 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'36, mlcod=10'36) v3)) start 2022-03-29T17:50:00.252 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,236 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'35 (0'0,10'35] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'35 mlcod 10'35 active append_log log((0'0,10'35], crt=10'35) {10'36 (10'35) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:58 2022-03-29T17:50:00.225471+0000 0 ObjectCleanRegions clean_offsets: [0~131072,196608~18446744073709355007], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.252 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,236 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'35 lcod 10'35 mlcod 10'35 active add_log_entry 10'36 (10'35) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:58 2022-03-29T17:50:00.225471+0000 0 ObjectCleanRegions clean_offsets: [0~131072,196608~18446744073709355007], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.253 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,236 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'36 lcod 10'35 mlcod 10'35 active append_log approx pg log length = 36 2022-03-29T17:50:00.253 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,236 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'36 lcod 10'35 mlcod 10'35 active append_log transaction_applied = 1 2022-03-29T17:50:00.253 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,236 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.253 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,236 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'36, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.254 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,236 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,237 [shard 0] osd - client_request(id=59, detail=m=[osd_op(client.4125.0:59 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,237 [shard 0] osd - client_request(id=59, detail=m=[osd_op(client.4125.0:59 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,237 [shard 0] osd - client_request(id=59, detail=m=[osd_op(client.4125.0:59 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,237 [shard 0] osd - OpSequencer::start_op: op=59, last_started=58, last_unblocked=58, last_completed=56 2022-03-29T17:50:00.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,238 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,238 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:57 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,238 [shard 0] osd - client_request(id=59, detail=m=[osd_op(client.4125.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,238 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,238 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,238 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,238 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.260 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,238 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,238 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:00.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.260 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,239 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:00.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'35 lua=0'0 crt=10'36 lcod 10'35 mlcod 10'35 active+clean calc_trim_to_aggressive limit = 10'35 2022-03-29T17:50:00.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'35 lua=0'0 crt=10'36 lcod 10'35 mlcod 10'35 active+clean append_log log((0'0,10'36], crt=10'36) {10'37 (10'36) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:59 2022-03-29T17:50:00.225501+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'35 lua=0'0 crt=10'36 lcod 10'35 mlcod 10'35 active+clean add_log_entry 10'37 (10'36) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:59 2022-03-29T17:50:00.225501+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'35 lua=0'0 crt=10'37 lcod 10'35 mlcod 10'35 active+clean append_log approx pg log length = 37 2022-03-29T17:50:00.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'35 lua=0'0 crt=10'37 lcod 10'35 mlcod 10'35 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,239 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'37, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,240 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,240 [shard 0] osd - OpSequencer::finish_op_in_order: op=57, last_started=59, last_unblocked=59, last_completed=56 2022-03-29T17:50:00.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,240 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4125.0:57 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.265 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,241 [shard 0] osd - replicated_request(id=37, detail=RepRequest(from=1 req=osd_repop(client.4125.0:59 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'37, mlcod=10'37) v3)) start 2022-03-29T17:50:00.265 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,241 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'36 (0'0,10'36] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'36 mlcod 10'36 active append_log log((0'0,10'36], crt=10'36) {10'37 (10'36) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:59 2022-03-29T17:50:00.225501+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.265 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,241 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'36 lcod 10'36 mlcod 10'36 active add_log_entry 10'37 (10'36) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:59 2022-03-29T17:50:00.225501+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.265 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,241 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'37 lcod 10'36 mlcod 10'36 active append_log approx pg log length = 37 2022-03-29T17:50:00.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,241 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'37 lcod 10'36 mlcod 10'36 active append_log transaction_applied = 1 2022-03-29T17:50:00.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,241 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,241 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'37, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,241 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,242 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:58 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 131072~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,243 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,243 [shard 0] osd - OpSequencer::finish_op_in_order: op=58, last_started=59, last_unblocked=59, last_completed=57 2022-03-29T17:50:00.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,243 [shard 0] osd - client_request(id=58, detail=m=[osd_op(client.4125.0:58 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 131072~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,243 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,244 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,244 [shard 0] osd - OpSequencer::finish_op_in_order: op=59, last_started=59, last_unblocked=59, last_completed=58 2022-03-29T17:50:00.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,244 [shard 0] osd - client_request(id=59, detail=m=[osd_op(client.4125.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,279 [shard 0] osd - client_request(id=60, detail=m=[osd_op(client.4125.0:60 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,279 [shard 0] osd - client_request(id=60, detail=m=[osd_op(client.4125.0:60 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,279 [shard 0] osd - client_request(id=60, detail=m=[osd_op(client.4125.0:60 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,279 [shard 0] osd - OpSequencer::start_op: op=60, last_started=59, last_unblocked=59, last_completed=59 2022-03-29T17:50:00.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - client_request(id=60, detail=m=[osd_op(client.4125.0:60 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'37 mlcod 10'37 active+clean calc_trim_to_aggressive limit = 10'37 2022-03-29T17:50:00.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'37 mlcod 10'37 active+clean append_log log((0'0,10'37], crt=10'37) {10'38 (10'34) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:60 2022-03-29T17:50:00.279468+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'37 lua=0'0 crt=10'37 lcod 10'37 mlcod 10'37 active+clean add_log_entry 10'38 (10'34) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:60 2022-03-29T17:50:00.279468+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'37 lua=0'0 crt=10'38 lcod 10'37 mlcod 10'37 active+clean append_log approx pg log length = 38 2022-03-29T17:50:00.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'37 lua=0'0 crt=10'38 lcod 10'37 mlcod 10'37 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'38, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,280 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,284 [shard 0] osd - replicated_request(id=38, detail=RepRequest(from=1 req=osd_repop(client.4125.0:60 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'38, mlcod=10'38) v3)) start 2022-03-29T17:50:00.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,284 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'37 (0'0,10'37] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'37 mlcod 10'37 active append_log log((0'0,10'37], crt=10'37) {10'38 (10'34) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:60 2022-03-29T17:50:00.279468+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,284 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'37 lcod 10'37 mlcod 10'37 active add_log_entry 10'38 (10'34) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:60 2022-03-29T17:50:00.279468+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,285 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'38 lcod 10'37 mlcod 10'37 active append_log approx pg log length = 38 2022-03-29T17:50:00.290 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,285 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'38 lcod 10'37 mlcod 10'37 active append_log transaction_applied = 1 2022-03-29T17:50:00.290 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,285 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.290 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,285 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'38, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.291 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,285 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,325 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:60 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,325 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - OpSequencer::finish_op_in_order: op=60, last_started=60, last_unblocked=60, last_completed=59 2022-03-29T17:50:00.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - client_request(id=60, detail=m=[osd_op(client.4125.0:60 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - client_request(id=61, detail=m=[osd_op(client.4125.0:61 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - client_request(id=61, detail=m=[osd_op(client.4125.0:61 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - client_request(id=61, detail=m=[osd_op(client.4125.0:61 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - OpSequencer::start_op: op=61, last_started=60, last_unblocked=60, last_completed=60 2022-03-29T17:50:00.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - client_request(id=61, detail=m=[osd_op(client.4125.0:61 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,326 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:50:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,327 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:50:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,327 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,328 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:50:00.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'38 mlcod 10'38 active+clean calc_trim_to_aggressive limit = 10'38 2022-03-29T17:50:00.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'38 mlcod 10'38 active+clean append_log log((0'0,10'38], crt=10'38) {10'39 (10'37) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:61 2022-03-29T17:50:00.326368+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'38 lua=0'0 crt=10'38 lcod 10'38 mlcod 10'38 active+clean add_log_entry 10'39 (10'37) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:61 2022-03-29T17:50:00.326368+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'38 lua=0'0 crt=10'39 lcod 10'38 mlcod 10'38 active+clean append_log approx pg log length = 39 2022-03-29T17:50:00.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'38 lua=0'0 crt=10'39 lcod 10'38 mlcod 10'38 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'39, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,328 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.368 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,366 [shard 0] osd - replicated_request(id=39, detail=RepRequest(from=1 req=osd_repop(client.4125.0:61 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'39, mlcod=10'39) v3)) start 2022-03-29T17:50:00.369 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,366 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'38 (0'0,10'38] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'38 mlcod 10'38 active append_log log((0'0,10'38], crt=10'38) {10'39 (10'37) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:61 2022-03-29T17:50:00.326368+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.369 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,366 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'38 lcod 10'38 mlcod 10'38 active add_log_entry 10'39 (10'37) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:61 2022-03-29T17:50:00.326368+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.369 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,367 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'39 lcod 10'38 mlcod 10'38 active append_log approx pg log length = 39 2022-03-29T17:50:00.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,367 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'39 lcod 10'38 mlcod 10'38 active append_log transaction_applied = 1 2022-03-29T17:50:00.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,367 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,367 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'39, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,367 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,407 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:61 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,408 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,408 [shard 0] osd - OpSequencer::finish_op_in_order: op=61, last_started=61, last_unblocked=61, last_completed=60 2022-03-29T17:50:00.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,408 [shard 0] osd - client_request(id=61, detail=m=[osd_op(client.4125.0:61 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,408 [shard 0] osd - client_request(id=62, detail=m=[osd_op(client.4125.0:62 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,408 [shard 0] osd - client_request(id=62, detail=m=[osd_op(client.4125.0:62 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - client_request(id=62, detail=m=[osd_op(client.4125.0:62 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - OpSequencer::start_op: op=62, last_started=61, last_unblocked=61, last_completed=61 2022-03-29T17:50:00.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - client_request(id=62, detail=m=[osd_op(client.4125.0:62 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:00.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:50:00.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,409 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.422 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,410 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:50:00.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:00.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:50:00.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.424 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,410 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'39 mlcod 10'39 active+clean calc_trim_to_aggressive limit = 10'39 2022-03-29T17:50:00.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'39 mlcod 10'39 active+clean append_log log((0'0,10'39], crt=10'39) {10'40 (10'39) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:62 2022-03-29T17:50:00.408606+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'39 lua=0'0 crt=10'39 lcod 10'39 mlcod 10'39 active+clean add_log_entry 10'40 (10'39) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:62 2022-03-29T17:50:00.408606+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'39 lua=0'0 crt=10'40 lcod 10'39 mlcod 10'39 active+clean append_log approx pg log length = 40 2022-03-29T17:50:00.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'39 lua=0'0 crt=10'40 lcod 10'39 mlcod 10'39 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'40, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,410 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.451 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,448 [shard 0] osd - replicated_request(id=40, detail=RepRequest(from=1 req=osd_repop(client.4125.0:62 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'40, mlcod=10'40) v3)) start 2022-03-29T17:50:00.452 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,448 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'39 (0'0,10'39] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'39 mlcod 10'39 active append_log log((0'0,10'39], crt=10'39) {10'40 (10'39) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:62 2022-03-29T17:50:00.408606+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.453 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,448 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'39 lcod 10'39 mlcod 10'39 active add_log_entry 10'40 (10'39) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:62 2022-03-29T17:50:00.408606+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.453 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,448 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'40 lcod 10'39 mlcod 10'39 active append_log approx pg log length = 40 2022-03-29T17:50:00.453 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,448 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'40 lcod 10'39 mlcod 10'39 active append_log transaction_applied = 1 2022-03-29T17:50:00.453 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,448 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.453 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,449 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'40, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.454 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,449 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,489 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:62 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - OpSequencer::finish_op_in_order: op=62, last_started=62, last_unblocked=62, last_completed=61 2022-03-29T17:50:00.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - client_request(id=62, detail=m=[osd_op(client.4125.0:62 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - client_request(id=63, detail=m=[osd_op(client.4125.0:63 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - client_request(id=63, detail=m=[osd_op(client.4125.0:63 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - client_request(id=63, detail=m=[osd_op(client.4125.0:63 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - OpSequencer::start_op: op=63, last_started=62, last_unblocked=62, last_completed=62 2022-03-29T17:50:00.513 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.513 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,490 [shard 0] osd - client_request(id=63, detail=m=[osd_op(client.4125.0:63 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.513 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.514 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.514 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.514 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.514 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.515 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.515 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.515 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:00.515 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:50:00.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:00.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:63 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,491 [shard 0] osd - OpSequencer::finish_op_in_order: op=63, last_started=63, last_unblocked=63, last_completed=62 2022-03-29T17:50:00.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - client_request(id=63, detail=m=[osd_op(client.4125.0:63 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - client_request(id=64, detail=m=[osd_op(client.4125.0:64 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - client_request(id=64, detail=m=[osd_op(client.4125.0:64 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - client_request(id=64, detail=m=[osd_op(client.4125.0:64 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - OpSequencer::start_op: op=64, last_started=63, last_unblocked=63, last_completed=63 2022-03-29T17:50:00.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - client_request(id=64, detail=m=[osd_op(client.4125.0:64 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op read 2022-03-29T17:50:00.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - handling op read on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - read: data length: 1 2022-03-29T17:50:00.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,492 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:64 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - OpSequencer::finish_op_in_order: op=64, last_started=64, last_unblocked=64, last_completed=63 2022-03-29T17:50:00.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - client_request(id=64, detail=m=[osd_op(client.4125.0:64 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - client_request(id=65, detail=m=[osd_op(client.4125.0:65 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - client_request(id=65, detail=m=[osd_op(client.4125.0:65 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - client_request(id=65, detail=m=[osd_op(client.4125.0:65 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - OpSequencer::start_op: op=65, last_started=64, last_unblocked=64, last_completed=64 2022-03-29T17:50:00.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - client_request(id=65, detail=m=[osd_op(client.4125.0:65 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.525 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.525 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:50:00.525 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,493 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - read: data length: 16 2022-03-29T17:50:00.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.527 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:65 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.527 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.527 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - OpSequencer::finish_op_in_order: op=65, last_started=65, last_unblocked=65, last_completed=64 2022-03-29T17:50:00.528 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - client_request(id=65, detail=m=[osd_op(client.4125.0:65 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.528 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - client_request(id=66, detail=m=[osd_op(client.4125.0:66 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - client_request(id=66, detail=m=[osd_op(client.4125.0:66 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - client_request(id=66, detail=m=[osd_op(client.4125.0:66 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - OpSequencer::start_op: op=66, last_started=65, last_unblocked=65, last_completed=65 2022-03-29T17:50:00.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - client_request(id=66, detail=m=[osd_op(client.4125.0:66 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,494 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:50:00.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:50:00.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:50:00.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:00.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - OpSequencer::finish_op_in_order: op=66, last_started=66, last_unblocked=66, last_completed=65 2022-03-29T17:50:00.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - client_request(id=66, detail=m=[osd_op(client.4125.0:66 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - client_request(id=67, detail=m=[osd_op(client.4125.0:67 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - client_request(id=67, detail=m=[osd_op(client.4125.0:67 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - client_request(id=67, detail=m=[osd_op(client.4125.0:67 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,495 [shard 0] osd - OpSequencer::start_op: op=67, last_started=66, last_unblocked=66, last_completed=66 2022-03-29T17:50:00.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - client_request(id=67, detail=m=[osd_op(client.4125.0:67 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'40 mlcod 10'40 active+clean calc_trim_to_aggressive limit = 10'40 2022-03-29T17:50:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'40 mlcod 10'40 active+clean append_log log((0'0,10'40], crt=10'40) {10'41 (10'38) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:67 2022-03-29T17:50:00.495514+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'40 lcod 10'40 mlcod 10'40 active+clean add_log_entry 10'41 (10'38) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:67 2022-03-29T17:50:00.495514+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'41 lcod 10'40 mlcod 10'40 active+clean append_log approx pg log length = 41 2022-03-29T17:50:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'41 lcod 10'40 mlcod 10'40 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'41, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,496 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - client_request(id=68, detail=m=[osd_op(client.4125.0:68 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - client_request(id=68, detail=m=[osd_op(client.4125.0:68 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - client_request(id=68, detail=m=[osd_op(client.4125.0:68 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - OpSequencer::start_op: op=68, last_started=67, last_unblocked=67, last_completed=66 2022-03-29T17:50:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - client_request(id=68, detail=m=[osd_op(client.4125.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,497 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,498 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'41 lcod 10'40 mlcod 10'40 active+clean calc_trim_to_aggressive limit = 10'40 2022-03-29T17:50:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,498 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'41 lcod 10'40 mlcod 10'40 active+clean append_log log((0'0,10'41], crt=10'41) {10'42 (10'41) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:68 2022-03-29T17:50:00.495546+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,498 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'41 lcod 10'40 mlcod 10'40 active+clean add_log_entry 10'42 (10'41) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:68 2022-03-29T17:50:00.495546+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,498 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'42 lcod 10'40 mlcod 10'40 active+clean append_log approx pg log length = 42 2022-03-29T17:50:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,498 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'42 lcod 10'40 mlcod 10'40 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,498 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,498 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'42, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,498 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - client_request(id=69, detail=m=[osd_op(client.4125.0:69 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - client_request(id=69, detail=m=[osd_op(client.4125.0:69 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - client_request(id=69, detail=m=[osd_op(client.4125.0:69 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - OpSequencer::start_op: op=69, last_started=68, last_unblocked=68, last_completed=66 2022-03-29T17:50:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - client_request(id=69, detail=m=[osd_op(client.4125.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'42 lcod 10'40 mlcod 10'40 active+clean calc_trim_to_aggressive limit = 10'40 2022-03-29T17:50:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'42 lcod 10'40 mlcod 10'40 active+clean append_log log((0'0,10'42], crt=10'42) {10'43 (10'42) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:69 2022-03-29T17:50:00.495567+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'42 lcod 10'40 mlcod 10'40 active+clean add_log_entry 10'43 (10'42) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:69 2022-03-29T17:50:00.495567+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'43 lcod 10'40 mlcod 10'40 active+clean append_log approx pg log length = 43 2022-03-29T17:50:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'43 lcod 10'40 mlcod 10'40 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,499 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'43, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,500 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,500 [shard 0] osd - client_request(id=70, detail=m=[osd_op(client.4125.0:70 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,500 [shard 0] osd - client_request(id=70, detail=m=[osd_op(client.4125.0:70 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.549 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,500 [shard 0] osd - replicated_request(id=41, detail=RepRequest(from=1 req=osd_repop(client.4125.0:67 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'41, mlcod=10'41) v3)) start 2022-03-29T17:50:00.549 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,501 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'40 (0'0,10'40] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'40 mlcod 10'40 active append_log log((0'0,10'40], crt=10'40) {10'41 (10'38) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:67 2022-03-29T17:50:00.495514+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.549 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,501 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'40 lcod 10'40 mlcod 10'40 active add_log_entry 10'41 (10'38) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:67 2022-03-29T17:50:00.495514+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,501 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'41 lcod 10'40 mlcod 10'40 active append_log approx pg log length = 41 2022-03-29T17:50:00.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,501 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'41 lcod 10'40 mlcod 10'40 active append_log transaction_applied = 1 2022-03-29T17:50:00.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,501 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,501 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'41, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.551 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,501 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.551 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,501 [shard 0] osd - replicated_request(id=42, detail=RepRequest(from=1 req=osd_repop(client.4125.0:68 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'42, mlcod=10'42) v3)) start 2022-03-29T17:50:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,502 [shard 0] osd - client_request(id=70, detail=m=[osd_op(client.4125.0:70 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,502 [shard 0] osd - OpSequencer::start_op: op=70, last_started=69, last_unblocked=69, last_completed=66 2022-03-29T17:50:00.553 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,503 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'41 (0'0,10'41] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'41 mlcod 10'41 active append_log log((0'0,10'41], crt=10'41) {10'42 (10'41) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:68 2022-03-29T17:50:00.495546+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.554 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,503 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'41 lcod 10'41 mlcod 10'41 active add_log_entry 10'42 (10'41) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:68 2022-03-29T17:50:00.495546+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.554 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,503 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'42 lcod 10'41 mlcod 10'41 active append_log approx pg log length = 42 2022-03-29T17:50:00.555 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,503 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'42 lcod 10'41 mlcod 10'41 active append_log transaction_applied = 1 2022-03-29T17:50:00.555 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,503 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.555 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,503 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'42, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.556 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,503 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.556 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,503 [shard 0] osd - replicated_request(id=43, detail=RepRequest(from=1 req=osd_repop(client.4125.0:69 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'43, mlcod=10'43) v3)) start 2022-03-29T17:50:00.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - client_request(id=70, detail=m=[osd_op(client.4125.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'43 lcod 10'40 mlcod 10'40 active+clean calc_trim_to_aggressive limit = 10'40 2022-03-29T17:50:00.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'43 lcod 10'40 mlcod 10'40 active+clean append_log log((0'0,10'43], crt=10'43) {10'44 (10'43) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:70 2022-03-29T17:50:00.495582+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'43 lcod 10'40 mlcod 10'40 active+clean add_log_entry 10'44 (10'43) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:70 2022-03-29T17:50:00.495582+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'44 lcod 10'40 mlcod 10'40 active+clean append_log approx pg log length = 44 2022-03-29T17:50:00.565 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'42 (0'0,10'42] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'42 mlcod 10'42 active append_log log((0'0,10'42], crt=10'42) {10'43 (10'42) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:69 2022-03-29T17:50:00.495567+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.566 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'42 lcod 10'42 mlcod 10'42 active add_log_entry 10'43 (10'42) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:69 2022-03-29T17:50:00.495567+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.566 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'43 lcod 10'42 mlcod 10'42 active append_log approx pg log length = 43 2022-03-29T17:50:00.567 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'43 lcod 10'42 mlcod 10'42 active append_log transaction_applied = 1 2022-03-29T17:50:00.567 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.567 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,504 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'43, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.568 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,505 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,505 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'40 lua=0'0 crt=10'44 lcod 10'40 mlcod 10'40 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,505 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,505 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'44, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,505 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.572 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,507 [shard 0] osd - replicated_request(id=44, detail=RepRequest(from=1 req=osd_repop(client.4125.0:70 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'44, mlcod=10'44) v3)) start 2022-03-29T17:50:00.573 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,507 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'43 (0'0,10'43] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'43 mlcod 10'43 active append_log log((0'0,10'43], crt=10'43) {10'44 (10'43) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:70 2022-03-29T17:50:00.495582+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.573 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,507 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'43 lcod 10'43 mlcod 10'43 active add_log_entry 10'44 (10'43) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:70 2022-03-29T17:50:00.495582+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.573 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,507 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'44 lcod 10'43 mlcod 10'43 active append_log approx pg log length = 44 2022-03-29T17:50:00.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,507 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'44 lcod 10'43 mlcod 10'43 active append_log transaction_applied = 1 2022-03-29T17:50:00.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,507 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,507 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'44, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,507 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,508 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:67 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,509 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,509 [shard 0] osd - OpSequencer::finish_op_in_order: op=67, last_started=70, last_unblocked=70, last_completed=66 2022-03-29T17:50:00.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,509 [shard 0] osd - client_request(id=67, detail=m=[osd_op(client.4125.0:67 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,509 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,509 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,509 [shard 0] osd - OpSequencer::finish_op_in_order: op=68, last_started=70, last_unblocked=70, last_completed=67 2022-03-29T17:50:00.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,509 [shard 0] osd - client_request(id=68, detail=m=[osd_op(client.4125.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,510 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,510 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,510 [shard 0] osd - OpSequencer::finish_op_in_order: op=69, last_started=70, last_unblocked=70, last_completed=68 2022-03-29T17:50:00.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,510 [shard 0] osd - client_request(id=69, detail=m=[osd_op(client.4125.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,511 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,511 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,511 [shard 0] osd - OpSequencer::finish_op_in_order: op=70, last_started=70, last_unblocked=70, last_completed=69 2022-03-29T17:50:00.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,511 [shard 0] osd - client_request(id=70, detail=m=[osd_op(client.4125.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,549 [shard 0] osd - client_request(id=71, detail=m=[osd_op(client.4125.0:71 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,549 [shard 0] osd - client_request(id=71, detail=m=[osd_op(client.4125.0:71 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,550 [shard 0] osd - client_request(id=71, detail=m=[osd_op(client.4125.0:71 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,550 [shard 0] osd - OpSequencer::start_op: op=71, last_started=70, last_unblocked=70, last_completed=70 2022-03-29T17:50:00.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,550 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,550 [shard 0] osd - client_request(id=71, detail=m=[osd_op(client.4125.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,550 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,550 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,550 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:00.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'44 mlcod 10'44 active+clean calc_trim_to_aggressive limit = 10'44 2022-03-29T17:50:00.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'44 mlcod 10'44 active+clean append_log log((0'0,10'44], crt=10'44) {10'45 (10'40) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:71 2022-03-29T17:50:00.549473+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'44 lua=0'0 crt=10'44 lcod 10'44 mlcod 10'44 active+clean add_log_entry 10'45 (10'40) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:71 2022-03-29T17:50:00.549473+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'44 lua=0'0 crt=10'45 lcod 10'44 mlcod 10'44 active+clean append_log approx pg log length = 45 2022-03-29T17:50:00.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'44 lua=0'0 crt=10'45 lcod 10'44 mlcod 10'44 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'45, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - client_request(id=72, detail=m=[osd_op(client.4125.0:72 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,551 [shard 0] osd - client_request(id=72, detail=m=[osd_op(client.4125.0:72 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,552 [shard 0] osd - replicated_request(id=45, detail=RepRequest(from=1 req=osd_repop(client.4125.0:71 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'45, mlcod=10'45) v3)) start 2022-03-29T17:50:00.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,552 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'44 (0'0,10'44] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'44 mlcod 10'44 active append_log log((0'0,10'44], crt=10'44) {10'45 (10'40) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:71 2022-03-29T17:50:00.549473+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,552 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'44 lcod 10'44 mlcod 10'44 active add_log_entry 10'45 (10'40) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:71 2022-03-29T17:50:00.549473+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,552 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'45 lcod 10'44 mlcod 10'44 active append_log approx pg log length = 45 2022-03-29T17:50:00.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,552 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'45 lcod 10'44 mlcod 10'44 active append_log transaction_applied = 1 2022-03-29T17:50:00.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,552 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,553 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'45, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,553 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,554 [shard 0] osd - client_request(id=72, detail=m=[osd_op(client.4125.0:72 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,554 [shard 0] osd - OpSequencer::start_op: op=72, last_started=71, last_unblocked=71, last_completed=70 2022-03-29T17:50:00.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,554 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,554 [shard 0] osd - client_request(id=72, detail=m=[osd_op(client.4125.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 196608~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,554 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,554 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - client_request(id=73, detail=m=[osd_op(client.4125.0:73 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - client_request(id=73, detail=m=[osd_op(client.4125.0:73 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:00.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'44 lua=0'0 crt=10'45 lcod 10'44 mlcod 10'44 active+clean calc_trim_to_aggressive limit = 10'44 2022-03-29T17:50:00.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'44 lua=0'0 crt=10'45 lcod 10'44 mlcod 10'44 active+clean append_log log((0'0,10'45], crt=10'45) {10'46 (10'45) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:72 2022-03-29T17:50:00.549523+0000 0 ObjectCleanRegions clean_offsets: [0~196608,262144~18446744073709289471], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'44 lua=0'0 crt=10'45 lcod 10'44 mlcod 10'44 active+clean add_log_entry 10'46 (10'45) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:72 2022-03-29T17:50:00.549523+0000 0 ObjectCleanRegions clean_offsets: [0~196608,262144~18446744073709289471], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'44 lua=0'0 crt=10'46 lcod 10'44 mlcod 10'44 active+clean append_log approx pg log length = 46 2022-03-29T17:50:00.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'44 lua=0'0 crt=10'46 lcod 10'44 mlcod 10'44 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'46, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,555 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,556 [shard 0] osd - replicated_request(id=46, detail=RepRequest(from=1 req=osd_repop(client.4125.0:72 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'46, mlcod=10'46) v3)) start 2022-03-29T17:50:00.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,556 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'45 (0'0,10'45] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'45 mlcod 10'45 active append_log log((0'0,10'45], crt=10'45) {10'46 (10'45) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:72 2022-03-29T17:50:00.549523+0000 0 ObjectCleanRegions clean_offsets: [0~196608,262144~18446744073709289471], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,556 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'45 lcod 10'45 mlcod 10'45 active add_log_entry 10'46 (10'45) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:72 2022-03-29T17:50:00.549523+0000 0 ObjectCleanRegions clean_offsets: [0~196608,262144~18446744073709289471], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,556 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'46 lcod 10'45 mlcod 10'45 active append_log approx pg log length = 46 2022-03-29T17:50:00.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,556 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'46 lcod 10'45 mlcod 10'45 active append_log transaction_applied = 1 2022-03-29T17:50:00.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,556 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,556 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'46, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,557 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,557 [shard 0] osd - client_request(id=73, detail=m=[osd_op(client.4125.0:73 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,557 [shard 0] osd - OpSequencer::start_op: op=73, last_started=72, last_unblocked=72, last_completed=70 2022-03-29T17:50:00.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,558 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,558 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,558 [shard 0] osd - client_request(id=74, detail=m=[osd_op(client.4125.0:74 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,558 [shard 0] osd - client_request(id=74, detail=m=[osd_op(client.4125.0:74 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,558 [shard 0] osd - client_request(id=73, detail=m=[osd_op(client.4125.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 262144~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,558 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:00.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'46 lcod 10'45 mlcod 10'45 active+clean calc_trim_to_aggressive limit = 10'45 2022-03-29T17:50:00.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'46 lcod 10'45 mlcod 10'45 active+clean append_log log((0'0,10'46], crt=10'46) {10'47 (10'46) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:73 2022-03-29T17:50:00.549571+0000 0 ObjectCleanRegions clean_offsets: [0~262144,327680~18446744073709223935], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'46 lcod 10'45 mlcod 10'45 active+clean add_log_entry 10'47 (10'46) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:73 2022-03-29T17:50:00.549571+0000 0 ObjectCleanRegions clean_offsets: [0~262144,327680~18446744073709223935], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'47 lcod 10'45 mlcod 10'45 active+clean append_log approx pg log length = 47 2022-03-29T17:50:00.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'47 lcod 10'45 mlcod 10'45 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,559 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'47, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,560 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,560 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,560 [shard 0] osd - OpSequencer::finish_op_in_order: op=71, last_started=73, last_unblocked=73, last_completed=70 2022-03-29T17:50:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - replicated_request(id=47, detail=RepRequest(from=1 req=osd_repop(client.4125.0:73 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'47, mlcod=10'47) v3)) start 2022-03-29T17:50:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'46 (0'0,10'46] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'46 mlcod 10'46 active append_log log((0'0,10'46], crt=10'46) {10'47 (10'46) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:73 2022-03-29T17:50:00.549571+0000 0 ObjectCleanRegions clean_offsets: [0~262144,327680~18446744073709223935], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'46 lcod 10'46 mlcod 10'46 active add_log_entry 10'47 (10'46) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:73 2022-03-29T17:50:00.549571+0000 0 ObjectCleanRegions clean_offsets: [0~262144,327680~18446744073709223935], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'47 lcod 10'46 mlcod 10'46 active append_log approx pg log length = 47 2022-03-29T17:50:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'47 lcod 10'46 mlcod 10'46 active append_log transaction_applied = 1 2022-03-29T17:50:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'47, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,560 [shard 0] osd - client_request(id=74, detail=m=[osd_op(client.4125.0:74 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,561 [shard 0] osd - OpSequencer::start_op: op=74, last_started=73, last_unblocked=73, last_completed=71 2022-03-29T17:50:00.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,562 [shard 0] osd - client_request(id=71, detail=m=[osd_op(client.4125.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,562 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,562 [shard 0] osd - client_request(id=74, detail=m=[osd_op(client.4125.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,562 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,562 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,562 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,562 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.619 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,562 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,562 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:00.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.620 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,563 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:00.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'47 lcod 10'45 mlcod 10'45 active+clean calc_trim_to_aggressive limit = 10'45 2022-03-29T17:50:00.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'47 lcod 10'45 mlcod 10'45 active+clean append_log log((0'0,10'47], crt=10'47) {10'48 (10'47) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:74 2022-03-29T17:50:00.549593+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.622 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'47 lcod 10'45 mlcod 10'45 active+clean add_log_entry 10'48 (10'47) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:74 2022-03-29T17:50:00.549593+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.622 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'48 lcod 10'45 mlcod 10'45 active+clean append_log approx pg log length = 48 2022-03-29T17:50:00.622 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'45 lua=0'0 crt=10'48 lcod 10'45 mlcod 10'45 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.622 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'48, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,563 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,564 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 196608~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,564 [shard 0] osd - replicated_request(id=48, detail=RepRequest(from=1 req=osd_repop(client.4125.0:74 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'48, mlcod=10'48) v3)) start 2022-03-29T17:50:00.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,565 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'47 (0'0,10'47] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'47 mlcod 10'47 active append_log log((0'0,10'47], crt=10'47) {10'48 (10'47) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:74 2022-03-29T17:50:00.549593+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,565 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'47 lcod 10'47 mlcod 10'47 active add_log_entry 10'48 (10'47) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:74 2022-03-29T17:50:00.549593+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,565 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'48 lcod 10'47 mlcod 10'47 active append_log approx pg log length = 48 2022-03-29T17:50:00.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,565 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'48 lcod 10'47 mlcod 10'47 active append_log transaction_applied = 1 2022-03-29T17:50:00.626 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,565 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.626 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,565 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'48, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.626 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,565 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,566 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.630 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,566 [shard 0] osd - OpSequencer::finish_op_in_order: op=72, last_started=74, last_unblocked=74, last_completed=71 2022-03-29T17:50:00.630 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,566 [shard 0] osd - client_request(id=72, detail=m=[osd_op(client.4125.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 196608~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.630 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,566 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 262144~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.630 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,567 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,567 [shard 0] osd - OpSequencer::finish_op_in_order: op=73, last_started=74, last_unblocked=74, last_completed=72 2022-03-29T17:50:00.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,567 [shard 0] osd - client_request(id=73, detail=m=[osd_op(client.4125.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 262144~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,567 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,568 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,568 [shard 0] osd - OpSequencer::finish_op_in_order: op=74, last_started=74, last_unblocked=74, last_completed=73 2022-03-29T17:50:00.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,568 [shard 0] osd - client_request(id=74, detail=m=[osd_op(client.4125.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,601 [shard 0] osd - client_request(id=75, detail=m=[osd_op(client.4125.0:75 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,601 [shard 0] osd - client_request(id=75, detail=m=[osd_op(client.4125.0:75 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,601 [shard 0] osd - client_request(id=75, detail=m=[osd_op(client.4125.0:75 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,601 [shard 0] osd - OpSequencer::start_op: op=75, last_started=74, last_unblocked=74, last_completed=74 2022-03-29T17:50:00.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,601 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - client_request(id=75, detail=m=[osd_op(client.4125.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:00.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:00.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:00.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:00.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'48 mlcod 10'48 active+clean calc_trim_to_aggressive limit = 10'48 2022-03-29T17:50:00.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'48 mlcod 10'48 active+clean append_log log((0'0,10'48], crt=10'48) {10'49 (10'44) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:75 2022-03-29T17:50:00.601436+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'48 lua=0'0 crt=10'48 lcod 10'48 mlcod 10'48 active+clean add_log_entry 10'49 (10'44) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:75 2022-03-29T17:50:00.601436+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'48 lua=0'0 crt=10'49 lcod 10'48 mlcod 10'48 active+clean append_log approx pg log length = 49 2022-03-29T17:50:00.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'48 lua=0'0 crt=10'49 lcod 10'48 mlcod 10'48 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'49, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,602 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.638 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,608 [shard 0] osd - replicated_request(id=49, detail=RepRequest(from=1 req=osd_repop(client.4125.0:75 1.0 e10/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 10'49, mlcod=10'49) v3)) start 2022-03-29T17:50:00.638 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,608 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'48 (0'0,10'48] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'48 mlcod 10'48 active append_log log((0'0,10'48], crt=10'48) {10'49 (10'44) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:75 2022-03-29T17:50:00.601436+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.638 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,608 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'48 lcod 10'48 mlcod 10'48 active add_log_entry 10'49 (10'44) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:75 2022-03-29T17:50:00.601436+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.638 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,609 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'49 lcod 10'48 mlcod 10'48 active append_log approx pg log length = 49 2022-03-29T17:50:00.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,609 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'49 lcod 10'48 mlcod 10'48 active append_log transaction_applied = 1 2022-03-29T17:50:00.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,609 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,609 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'49, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,609 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,649 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:00.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,649 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:00.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - OpSequencer::finish_op_in_order: op=75, last_started=75, last_unblocked=75, last_completed=74 2022-03-29T17:50:00.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - client_request(id=75, detail=m=[osd_op(client.4125.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - client_request(id=76, detail=m=[osd_op(client.4125.0:76 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - client_request(id=76, detail=m=[osd_op(client.4125.0:76 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - client_request(id=76, detail=m=[osd_op(client.4125.0:76 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - OpSequencer::start_op: op=76, last_started=75, last_unblocked=75, last_completed=75 2022-03-29T17:50:00.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - client_request(id=76, detail=m=[osd_op(client.4125.0:76 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,650 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:00.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:50:00.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.659 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,651 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:00.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:50:00.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,651 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.660 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,651 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:50:00.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'49 mlcod 10'49 active+clean calc_trim_to_aggressive limit = 10'49 2022-03-29T17:50:00.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'49 mlcod 10'49 active+clean append_log log((0'0,10'49], crt=10'49) {10'50 (10'48) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:76 2022-03-29T17:50:00.650298+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'49 lcod 10'49 mlcod 10'49 active+clean add_log_entry 10'50 (10'48) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:76 2022-03-29T17:50:00.650298+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean append_log approx pg log length = 50 2022-03-29T17:50:00.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'50, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,652 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,690 [shard 0] osd - replicated_request(id=50, detail=RepRequest(from=1 req=osd_repop(client.4125.0:76 1.0 e10/8 1:30306672:devicehealth::main.db.0000000000000000:head v 10'50, mlcod=10'50) v3)) start 2022-03-29T17:50:00.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,690 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'49 (0'0,10'49] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'49 mlcod 10'49 active append_log log((0'0,10'49], crt=10'49) {10'50 (10'48) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:76 2022-03-29T17:50:00.650298+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,690 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'49 lcod 10'49 mlcod 10'49 active add_log_entry 10'50 (10'48) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:76 2022-03-29T17:50:00.650298+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,690 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active append_log approx pg log length = 50 2022-03-29T17:50:00.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,690 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active append_log transaction_applied = 1 2022-03-29T17:50:00.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,690 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,690 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 10'50, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,691 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:00.704 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,702 [shard 0] osd - handle_osd_map osd_map(10..11 src has 1..11) v4 2022-03-29T17:50:00.704 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,702 [shard 0] osd - handle_osd_map epochs [10..11], i have 7, src has [1..11] 2022-03-29T17:50:00.705 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,703 [shard 0] osd - handle_osd_map message skips epochs 8..9 2022-03-29T17:50:00.705 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,703 [shard 0] osd - osdmap_subscribe(8) 2022-03-29T17:50:00.707 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,705 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e11 {2.2=11,2022-03-29T17:50:00.701078+0000}) v2): starting 2022-03-29T17:50:00.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,705 [shard 0] osd - handle_pg_create: 2.2 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,705 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): start 2022-03-29T17:50:00.708 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,705 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e11 {2.2=11,2022-03-29T17:50:00.701078+0000}) v2): about to fork future 2022-03-29T17:50:00.709 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,705 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e11 {2.0=11,2022-03-29T17:50:00.701078+0000,2.1=11,2022-03-29T17:50:00.701078+0000}) v2): starting 2022-03-29T17:50:00.709 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,706 [shard 0] osd - handle_pg_create: 2.0 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.710 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,706 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): start 2022-03-29T17:50:00.710 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,706 [shard 0] osd - handle_pg_create: 2.1 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.710 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,706 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=? pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): start 2022-03-29T17:50:00.711 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,706 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e11 {2.0=11,2022-03-29T17:50:00.701078+0000,2.1=11,2022-03-29T17:50:00.701078+0000}) v2): about to fork future 2022-03-29T17:50:00.711 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,706 [shard 0] osd - evt epoch is 11, i have 7, will wait 2022-03-29T17:50:00.711 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,706 [shard 0] osd - osdmap_subscribe(7) 2022-03-29T17:50:00.711 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:50:00,706 [shard 0] monc - renew_subs - empty 2022-03-29T17:50:00.712 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,706 [shard 0] osd - handle_osd_map osd_map(8..11 src has 1..11) v4 2022-03-29T17:50:00.712 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,706 [shard 0] osd - handle_osd_map epochs [8..11], i have 7, src has [1..11] 2022-03-29T17:50:00.712 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,707 [shard 0] osd - evt epoch is 11, i have 9, will wait 2022-03-29T17:50:00.713 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,707 [shard 0] osd - osdmap_subscribe(9) 2022-03-29T17:50:00.713 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'rbd' created 2022-03-29T17:50:00.714 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,708 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=12990204181074666112 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:50:00.714 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,709 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:00.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,709 [shard 0] osd - handle_osd_map osd_map(9..11 src has 1..11) v4 2022-03-29T17:50:00.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,709 [shard 0] osd - handle_osd_map epochs [9..11], i have 9, src has [1..11] 2022-03-29T17:50:00.715 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,710 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:00.716 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,711 [shard 0] osd - osd.2: committed_osd_maps(8, 11) 2022-03-29T17:50:00.719 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,711 [shard 0] osd - osd.0: committed_osd_maps(10, 11) 2022-03-29T17:50:00.720 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,711 [shard 0] osd - osd.0: map e 11 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:00.720 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,712 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=15312318518825473467 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:50:00.720 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,712 [shard 0] osd - pg_advance_map(id=2, detail=PGAdvanceMap(pg=1.0 from=9 to=11)): start 2022-03-29T17:50:00.720 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,712 [shard 0] osd - pg_epoch 9 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:00.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,712 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active state: Started advmap 2022-03-29T17:50:00.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,712 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active check_recovery_sources no source osds () went down 2022-03-29T17:50:00.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,712 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:00.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,712 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active state: Started advmap 2022-03-29T17:50:00.722 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,712 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active check_recovery_sources no source osds () went down 2022-03-29T17:50:00.722 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,712 [shard 0] monc - get_auth_request(con=[osd.2(client) v2:172.21.15.138:6803/35579@52685 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:50:00.723 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,713 [shard 0] osd - osd.2: map e 11 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:00.723 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,712 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active activate_map 2022-03-29T17:50:00.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,713 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active activate_map: Not dirtying info: last_persisted is 9 while current is 11 2022-03-29T17:50:00.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,713 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,713 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.725 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,713 [shard 0] osd - osd.2: now active 2022-03-29T17:50:00.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,713 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): got map 11 2022-03-29T17:50:00.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,713 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=? pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): got map 11 2022-03-29T17:50:00.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,714 [shard 0] osd - Creating 2.0 2022-03-29T17:50:00.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,714 [shard 0] osd - pg_advance_map(id=2, detail=PGAdvanceMap(pg=1.0 from=9 to=11)): complete 2022-03-29T17:50:00.727 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,714 [shard 0] osd - osd.0: now active 2022-03-29T17:50:00.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,714 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): got map 11 2022-03-29T17:50:00.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,714 [shard 0] osd - Creating 2.2 2022-03-29T17:50:00.728 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,714 [shard 0] monc - get_auth_request(con=[osd.0(client) v2:172.21.15.138:6805/35577@54730 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:50:00.729 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:00.729 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,715 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:00.729 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 1} acting {2, 1} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.729 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:00.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:50:00.731 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:00.731 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,715 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:00.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:00.732 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:50:00.732 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:00.733 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:00.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:50:00.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 7.348740578s + 0.000000000s) 2022-03-29T17:50:00.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_advance_map(id=1, detail=PGAdvanceMap(pg=2.0 from=11 to=11 do_init)): start 2022-03-29T17:50:00.735 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000965 0 0.000000 2022-03-29T17:50:00.735 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,715 [shard 0] osd - Exiting state: Initial, entered at 1648576200.7150025, 0.0 spent on 0 events 2022-03-29T17:50:00.735 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:00.736 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:00.736 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:00.736 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:00.736 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:00.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,715 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:50:00.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 7.325015068s + 0.000000000s) 2022-03-29T17:50:00.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_advance_map(id=3, detail=PGAdvanceMap(pg=2.2 from=11 to=11 do_init)): start 2022-03-29T17:50:00.739 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000951 0 0.000000 2022-03-29T17:50:00.740 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Exiting state: Initial, entered at 1648576200.7154095, 0.0 spent on 0 events 2022-03-29T17:50:00.740 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:00.740 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:00.741 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:00.741 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:00.741 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:00.741 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:00.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:00.742 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000156 1 2022-03-29T17:50:00.716537+0000 2022-03-29T17:50:00.742 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Exiting state: Reset, entered at 1648576200.716411, 1648576200.7165372 spent on 1 events 2022-03-29T17:50:00.743 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:00.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:00.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:00.747 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000666 1 2022-03-29T17:50:00.716658+0000 2022-03-29T17:50:00.747 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Exiting state: Reset, entered at 1648576200.716026, 1648576200.7166584 spent on 1 events 2022-03-29T17:50:00.747 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Entering state: Started 2022-03-29T17:50:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Entering state: Start 2022-03-29T17:50:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:50:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000056 0 0.000000 2022-03-29T17:50:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Exiting state: Start, entered at 1648576200.7167695, 0.0 spent on 0 events 2022-03-29T17:50:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:50:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:50:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:50:00.750 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:50:00.750 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.750 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,716 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.750 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:50:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,716 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:50:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering up_thru 0 < same_since 11, must notify monitor 2022-03-29T17:50:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:50:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:50:00.754 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - Entering state: Started 2022-03-29T17:50:00.755 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:00.755 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - Entering state: Start 2022-03-29T17:50:00.755 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:50:00.755 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000053 0 0.000000 2022-03-29T17:50:00.756 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - Exiting state: Start, entered at 1648576200.7171528, 0.0 spent on 0 events 2022-03-29T17:50:00.756 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:50:00.756 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:50:00.757 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:50:00.757 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:50:00.757 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.757 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:50:00.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:50:00.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering up_thru 0 < same_since 11, must notify monitor 2022-03-29T17:50:00.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:50:00.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:50:00.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:50:00.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:00.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:00.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:50:00.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:00.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:00.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:1 2022-03-29T17:50:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - Created 2.0 2022-03-29T17:50:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:50:00.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.767 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,717 [shard 0] osd - send_alive want=11 up_thru_wanted=0 2022-03-29T17:50:00.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,717 [shard 0] osd - send_alive up_thru_wanted=11 up_thru=0 2022-03-29T17:50:00.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): pg present 2022-03-29T17:50:00.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:1 2022-03-29T17:50:00.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - Created 2.2 2022-03-29T17:50:00.770 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,718 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:50:00.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.770 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,718 [shard 0] osd - send_alive want=11 up_thru_wanted=0 2022-03-29T17:50:00.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - send_alive up_thru_wanted=11 up_thru=0 2022-03-29T17:50:00.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): pg present 2022-03-29T17:50:00.771 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - Creating 2.1 2022-03-29T17:50:00.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,718 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 NullEvt +create_info for pg: 2.2 2022-03-29T17:50:00.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:00.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.774 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,719 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e11 {2.2=11,2022-03-29T17:50:00.701078+0000}) v2): sub events complete 2022-03-29T17:50:00.774 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,719 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e11 {2.2=11,2022-03-29T17:50:00.701078+0000}) v2): complete 2022-03-29T17:50:00.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:00.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:00.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.2 2022-03-29T17:50:00.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:50:00.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 NullEvt +create_info for pg: 2.0 2022-03-29T17:50:00.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,719 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/common/hobject.h:92:27: runtime error: left shift of negative value -1 2022-03-29T17:50:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/common/hobject.h:92:57: runtime error: left shift of negative value -1 2022-03-29T17:50:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,719 [shard 0] osd - pg_epoch 11 pg[2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:00.778 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,719 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:00.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:00.779 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.779 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:00.779 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:00.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - pg_advance_map(id=3, detail=PGAdvanceMap(pg=2.2 from=11 to=11 do_init)): complete 2022-03-29T17:50:00.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - pg_epoch 11 pg[2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 1} acting {2, 1} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:00.783 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:50:00.783 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:00.783 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:50:00.783 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 7.353797436s + 0.000000000s) 2022-03-29T17:50:00.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - pg_advance_map(id=2, detail=PGAdvanceMap(pg=2.1 from=11 to=11 do_init)): start 2022-03-29T17:50:00.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,720 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:00.784 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001149 0 0.000000 2022-03-29T17:50:00.785 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - Exiting state: Initial, entered at 1648576200.719919, 0.0 spent on 0 events 2022-03-29T17:50:00.786 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): complete 2022-03-29T17:50:00.787 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:00.790 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:00.790 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:00.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:00.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:00.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:00.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:00.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:00.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000666 1 2022-03-29T17:50:00.721760+0000 2022-03-29T17:50:00.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - Exiting state: Reset, entered at 1648576200.721119, 1648576200.7217605 spent on 1 events 2022-03-29T17:50:00.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:00.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - Entering state: Started 2022-03-29T17:50:00.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:00.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - Entering state: Start 2022-03-29T17:50:00.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:50:00.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000047 0 0.000000 2022-03-29T17:50:00.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - Exiting state: Start, entered at 1648576200.7218802, 0.0 spent on 0 events 2022-03-29T17:50:00.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:50:00.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,721 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:50:00.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:50:00.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,722 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:50:00.796 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-03-29T17:50:00.801 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.802 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,722 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:50:00.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:50:00.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering up_thru 0 < same_since 11, must notify monitor 2022-03-29T17:50:00.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:50:00.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:50:00.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:50:00.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:00.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:00.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:1 2022-03-29T17:50:00.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - Created 2.1 2022-03-29T17:50:00.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,722 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:50:00.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.807 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,722 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,722 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=? pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): pg present 2022-03-29T17:50:00.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.0 2022-03-29T17:50:00.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:50:00.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:00.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:00.809 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,723 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.? v2:172.21.15.138:6801/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6801/35578, peer_name=osd.?, cc=14137331708520870358 policy(lossy=false, server=false, standby=true, resetcheck=false) 2022-03-29T17:50:00.810 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,723 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.? v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:50:00.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - pg_advance_map(id=1, detail=PGAdvanceMap(pg=2.0 from=11 to=11 do_init)): complete 2022-03-29T17:50:00.810 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,723 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:00.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 NullEvt +create_info for pg: 2.1 2022-03-29T17:50:00.812 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.812 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,723 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=? pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:00.812 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.812 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - peering_event(id=1, detail=PeeringEvent(from=? pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): complete 2022-03-29T17:50:00.813 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:00.813 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,724 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e11 {2.0=11,2022-03-29T17:50:00.701078+0000,2.1=11,2022-03-29T17:50:00.701078+0000}) v2): sub events complete 2022-03-29T17:50:00.813 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,724 [shard 0] osd - compound_peering_request(id=1, detail=pg_create2(e11 {2.0=11,2022-03-29T17:50:00.701078+0000,2.1=11,2022-03-29T17:50:00.701078+0000}) v2): complete 2022-03-29T17:50:00.813 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,724 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.814 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.814 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - peering_event(id=3, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:00.814 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.1 2022-03-29T17:50:00.814 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:50:00.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:00.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,724 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:00.816 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,725 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=10183302347089209809 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:50:00.816 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,725 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> unknown.? -@64300] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:64300/0 2022-03-29T17:50:00.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,726 [shard 0] osd - pg_advance_map(id=2, detail=PGAdvanceMap(pg=2.1 from=11 to=11 do_init)): complete 2022-03-29T17:50:00.818 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,727 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,727 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.818 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,727 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.819 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,727 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.819 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,727 [shard 0] osd - peering_event(id=4, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:00.819 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,727 [shard 0] osd - peering_event(id=2, detail=PeeringEvent(from=? pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): complete 2022-03-29T17:50:00.819 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,727 [shard 0] monc - get_auth_request(con=[osd.2(cluster) v2:172.21.15.138:6802/35579@64300 >> osd.? v2:172.21.15.138:6801/35578], auth_method=0) 2022-03-29T17:50:00.826 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,728 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e11 {2.3=11,2022-03-29T17:50:00.701078+0000,2.4=11,2022-03-29T17:50:00.701078+0000,2.5=11,2022-03-29T17:50:00.701078+0000,2.6=11,2022-03-29T17:50:00.701078+0000,2.7=11,2022-03-29T17:50:00.701078+0000}) v2): starting 2022-03-29T17:50:00.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - handle_pg_create: 2.3 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): start 2022-03-29T17:50:00.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - handle_pg_create: 2.4 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=? pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): start 2022-03-29T17:50:00.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - handle_pg_create: 2.5 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=? pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): start 2022-03-29T17:50:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - handle_pg_create: 2.6 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=? pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): start 2022-03-29T17:50:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - handle_pg_create: 2.7 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,728 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=? pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): start 2022-03-29T17:50:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,728 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e11 {2.3=11,2022-03-29T17:50:00.701078+0000,2.4=11,2022-03-29T17:50:00.701078+0000,2.5=11,2022-03-29T17:50:00.701078+0000,2.6=11,2022-03-29T17:50:00.701078+0000,2.7=11,2022-03-29T17:50:00.701078+0000}) v2): about to fork future 2022-03-29T17:50:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,728 [shard 0] osd - evt epoch is 11, i have 10, will wait 2022-03-29T17:50:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,729 [shard 0] osd - osdmap_subscribe(10) 2022-03-29T17:50:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,729 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.? -@64300] UPDATE: peer_type=osd, policy(lossy=false server=false standby=true resetcheck=false) 2022-03-29T17:50:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,729 [shard 0] monc - get_auth_request(con=[osd.1(client) v2:172.21.15.138:6800/35578@51139 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:50:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,729 [shard 0] osd - handle_osd_map osd_map(10..11 src has 1..11) v4 2022-03-29T17:50:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,729 [shard 0] osd - handle_osd_map epochs [10..11], i have 10, src has [1..11] 2022-03-29T17:50:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,730 [shard 0] monc - added challenge on [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.? -@64300] 2022-03-29T17:50:00.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,730 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:00.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,731 [shard 0] osd - handle_authentication osd.2 has caps allow * 2022-03-29T17:50:00.831 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,731 [shard 0] osd - osd.1: committed_osd_maps(11, 11) 2022-03-29T17:50:00.831 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,732 [shard 0] osd - osd.1: map e 11 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_advance_map(id=4, detail=PGAdvanceMap(pg=1.0 from=10 to=11)): start 2022-03-29T17:50:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 10 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean state: Active advmap 2022-03-29T17:50:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean state: Started advmap 2022-03-29T17:50:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean activate_map 2022-03-29T17:50:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean state: Active: handling ActMap 2022-03-29T17:50:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean state: handle ActMap primary 2022-03-29T17:50:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean publish_stats_to_osd 10:10 2022-03-29T17:50:00.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'49 lua=0'0 crt=10'50 lcod 10'49 mlcod 10'49 active+clean activate_map: Not dirtying info: last_persisted is 10 while current is 11 2022-03-29T17:50:00.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,733 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,734 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:76 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,734 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,734 [shard 0] osd - OpSequencer::finish_op_in_order: op=76, last_started=76, last_unblocked=76, last_completed=75 2022-03-29T17:50:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,734 [shard 0] osd - client_request(id=76, detail=m=[osd_op(client.4125.0:76 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,735 [shard 0] osd - pg_advance_map(id=4, detail=PGAdvanceMap(pg=1.0 from=10 to=11)): complete 2022-03-29T17:50:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,735 [shard 0] osd - osd.1: now active 2022-03-29T17:50:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,735 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,735 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.2 from 0 query_epoch 11 query: query(info 0'0 epoch_sent 11))): start 2022-03-29T17:50:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,735 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=? pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): got map 11 2022-03-29T17:50:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,735 [shard 0] osd - Creating 2.5 2022-03-29T17:50:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,735 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=? pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): got map 11 2022-03-29T17:50:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,735 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=? pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): got map 11 2022-03-29T17:50:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,735 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.2 from 0 query_epoch 11 query: query(info 0'0 epoch_sent 11))): got map 11 2022-03-29T17:50:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,736 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.2 v2:172.21.15.138:6802/35579@64300] established: gs=0, pgs=1, cs=0, client_cookie=14137331708520870358, server_cookie=851851571006733966, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:50:00,736 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.2 from 0 query_epoch 11 query: query(info 0'0 epoch_sent 11))): pg absent, did not create 2022-03-29T17:50:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,736 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.2 from 0 query_epoch 11 query: query(info 0'0 epoch_sent 11))): OSDState is active 2022-03-29T17:50:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,736 [shard 0] osd - pg_epoch 11 pg[2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,736 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:00.839 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,736 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@64300 >> osd.1 v2:172.21.15.138:6801/35578] connected: gs=1, pgs=2, cs=0, client_cookie=14137331708520870358, server_cookie=851851571006733966, in_seq=0, out_seq=0, out_q=2 2022-03-29T17:50:00.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,736 [shard 0] osd - pg_epoch 11 pg[2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,736 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:00.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:50:00.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:00.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:50:00.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 7.432985783s + 0.000000000s) 2022-03-29T17:50:00.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_advance_map(id=5, detail=PGAdvanceMap(pg=2.5 from=11 to=11 do_init)): start 2022-03-29T17:50:00.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=? pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): got map 11 2022-03-29T17:50:00.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:50:00.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.2 from 0 query_epoch 11 query: query(info 0'0 epoch_sent 11))): complete 2022-03-29T17:50:00.887 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001203 0 0.000000 2022-03-29T17:50:00.887 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Exiting state: Initial, entered at 1648576200.7362695, 0.0 spent on 0 events 2022-03-29T17:50:00.888 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:00.888 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:00.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:00.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:00.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:00.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:00.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:00.889 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000118 1 2022-03-29T17:50:00.737618+0000 2022-03-29T17:50:00.889 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Exiting state: Reset, entered at 1648576200.7375238, 1648576200.7376187 spent on 1 events 2022-03-29T17:50:00.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:00.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Entering state: Started 2022-03-29T17:50:00.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:00.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Entering state: Start 2022-03-29T17:50:00.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:50:00.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000040 0 0.000000 2022-03-29T17:50:00.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Exiting state: Start, entered at 1648576200.7377052, 0.0 spent on 0 events 2022-03-29T17:50:00.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:50:00.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:50:00.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:50:00.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:50:00.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,737 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:50:00.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:50:00.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering up_thru 8 < same_since 11, must notify monitor 2022-03-29T17:50:00.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:50:00.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:50:00.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,737 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:50:00.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:00.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:00.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:1 2022-03-29T17:50:00.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - Created 2.5 2022-03-29T17:50:00.896 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,738 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:50:00.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.897 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,738 [shard 0] osd - send_alive want=11 up_thru_wanted=8 2022-03-29T17:50:00.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - send_alive up_thru_wanted=11 up_thru=8 2022-03-29T17:50:00.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=? pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): pg present 2022-03-29T17:50:00.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): got map 11 2022-03-29T17:50:00.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - client_request(id=77, detail=m=[osd_op(client.4125.0:77 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:00.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - client_request(id=77, detail=m=[osd_op(client.4125.0:77 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:00.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - Creating 2.6 2022-03-29T17:50:00.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,738 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:00.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - client_request(id=77, detail=m=[osd_op(client.4125.0:77 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:00.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - OpSequencer::start_op: op=77, last_started=76, last_unblocked=76, last_completed=76 2022-03-29T17:50:00.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:00.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 NullEvt +create_info for pg: 2.5 2022-03-29T17:50:00.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=? pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:00.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.900 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,739 [shard 0] osd - pg_epoch 11 pg[2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:00.901 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,739 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:00.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - pg_epoch 11 pg[2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:00.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:50:00.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:00.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:50:00.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 7.435591698s + 0.000000000s) 2022-03-29T17:50:00.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - pg_advance_map(id=6, detail=PGAdvanceMap(pg=2.6 from=11 to=11 do_init)): start 2022-03-29T17:50:00.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,739 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.5 2022-03-29T17:50:00.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:50:00.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:50:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000833 0 0.000000 2022-03-29T17:50:00.905 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Exiting state: Initial, entered at 1648576200.7393494, 0.0 spent on 0 events 2022-03-29T17:50:00.905 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:00.905 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:00.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:00.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:00.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:00.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:00.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:00.907 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000125 1 2022-03-29T17:50:00.740340+0000 2022-03-29T17:50:00.907 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Exiting state: Reset, entered at 1648576200.7402465, 1648576200.7403405 spent on 1 events 2022-03-29T17:50:00.907 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:00.907 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Entering state: Started 2022-03-29T17:50:00.908 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:00.908 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Entering state: Start 2022-03-29T17:50:00.908 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:50:00.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000056 0 0.000000 2022-03-29T17:50:00.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Exiting state: Start, entered at 1648576200.740435, 0.0 spent on 0 events 2022-03-29T17:50:00.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:50:00.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:50:00.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:50:00.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:50:00.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.911 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:50:00.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:50:00.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering up_thru 8 < same_since 11, must notify monitor 2022-03-29T17:50:00.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:50:00.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:50:00.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:50:00.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_advance_map(id=5, detail=PGAdvanceMap(pg=2.5 from=11 to=11 do_init)): complete 2022-03-29T17:50:00.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:00.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:00.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:1 2022-03-29T17:50:00.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - Created 2.6 2022-03-29T17:50:00.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:50:00.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.917 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,740 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,740 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=? pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): pg present 2022-03-29T17:50:00.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:00.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.0 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): start 2022-03-29T17:50:00.918 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,741 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - client_request(id=77, detail=m=[osd_op(client.4125.0:77 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:00.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:00.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=? pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): complete 2022-03-29T17:50:00.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - Creating 2.7 2022-03-29T17:50:00.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,741 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:00.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,742 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,742 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:50:00.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,742 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,742 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.0 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): got map 11 2022-03-29T17:50:00.921 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,743 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 NullEvt +create_info for pg: 2.6 2022-03-29T17:50:00.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=? pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:00.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,743 [shard 0] osd - pg_epoch 11 pg[2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:00.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,743 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:00.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:00.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - pg_epoch 11 pg[2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:00.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:50:00.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:00.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:50:00.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 7.439665318s + 0.000000000s) 2022-03-29T17:50:00.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - pg_advance_map(id=7, detail=PGAdvanceMap(pg=2.7 from=11 to=11 do_init)): start 2022-03-29T17:50:00.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:00.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,743 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:00.925 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:50:00,744 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.0 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): pg absent, did not create 2022-03-29T17:50:00.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.0 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): OSDState is active 2022-03-29T17:50:00.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:50:00.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.0 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): complete 2022-03-29T17:50:00.926 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000848 0 0.000000 2022-03-29T17:50:00.927 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Exiting state: Initial, entered at 1648576200.74341, 0.0 spent on 0 events 2022-03-29T17:50:00.927 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:00.927 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:00.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:00.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000125 1 2022-03-29T17:50:00.744402+0000 2022-03-29T17:50:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Exiting state: Reset, entered at 1648576200.7443042, 1648576200.744403 spent on 1 events 2022-03-29T17:50:00.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:00.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Entering state: Started 2022-03-29T17:50:00.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:00.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Entering state: Start 2022-03-29T17:50:00.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:50:00.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:50:00.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Exiting state: Start, entered at 1648576200.7444897, 0.0 spent on 0 events 2022-03-29T17:50:00.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:50:00.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:50:00.931 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:50:00.931 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:50:00.931 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.931 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:50:00.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:50:00.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering up_thru 8 < same_since 11, must notify monitor 2022-03-29T17:50:00.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:50:00.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:50:00.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:50:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:00.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:1 2022-03-29T17:50:00.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - Created 2.7 2022-03-29T17:50:00.935 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,744 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:50:00.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,744 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.935 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,745 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - pg_advance_map(id=6, detail=PGAdvanceMap(pg=2.6 from=11 to=11 do_init)): complete 2022-03-29T17:50:00.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=? pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): pg present 2022-03-29T17:50:00.936 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,745 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=? pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): complete 2022-03-29T17:50:00.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.937 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,745 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:00.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:00.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - Creating 2.4 2022-03-29T17:50:00.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:00.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:50:00.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:00.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.1 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): start 2022-03-29T17:50:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.6 2022-03-29T17:50:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:50:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,745 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:50:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,746 [shard 0] osd - pg_epoch 11 pg[2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,746 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - pg_epoch 11 pg[2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:50:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:50:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 7.442585468s + 0.000000000s) 2022-03-29T17:50:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - pg_advance_map(id=8, detail=PGAdvanceMap(pg=2.4 from=11 to=11 do_init)): start 2022-03-29T17:50:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 NullEvt +create_info for pg: 2.7 2022-03-29T17:50:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,746 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=? pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000578 0 0.000000 2022-03-29T17:50:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Exiting state: Initial, entered at 1648576200.746457, 0.0 spent on 0 events 2022-03-29T17:50:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000102 1 2022-03-29T17:50:00.747156+0000 2022-03-29T17:50:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Exiting state: Reset, entered at 1648576200.747074, 1648576200.7471569 spent on 1 events 2022-03-29T17:50:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Entering state: Started 2022-03-29T17:50:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Entering state: Start 2022-03-29T17:50:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:50:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000081 0 0.000000 2022-03-29T17:50:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Exiting state: Start, entered at 1648576200.7472334, 0.0 spent on 0 events 2022-03-29T17:50:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:50:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:50:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:50:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:50:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:50:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:50:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering up_thru 8 < same_since 11, must notify monitor 2022-03-29T17:50:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:50:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:50:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:50:00.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:1 2022-03-29T17:50:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - Created 2.4 2022-03-29T17:50:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:50:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,747 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - pg_advance_map(id=7, detail=PGAdvanceMap(pg=2.7 from=11 to=11 do_init)): complete 2022-03-29T17:50:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=? pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): pg present 2022-03-29T17:50:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,747 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:50:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,748 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,748 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:00.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:00.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:00.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:00.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'50 mlcod 10'50 active+clean calc_trim_to_aggressive limit = 10'50 2022-03-29T17:50:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=10'50 mlcod 10'50 active+clean append_log log((0'0,10'50], crt=10'50) {11'51 (10'50) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:77 2022-03-29T17:50:00.734860+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'50 lua=0'0 crt=10'50 lcod 10'50 mlcod 10'50 active+clean add_log_entry 11'51 (10'50) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:77 2022-03-29T17:50:00.734860+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'50 lua=0'0 crt=11'51 lcod 10'50 mlcod 10'50 active+clean append_log approx pg log length = 51 2022-03-29T17:50:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=10'50 lua=0'0 crt=11'51 lcod 10'50 mlcod 10'50 active+clean append_log transaction_applied = 1 2022-03-29T17:50:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:00.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'51, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:00.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.7 2022-03-29T17:50:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:50:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:50:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,748 [shard 0] osd - Creating 2.3 2022-03-29T17:50:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.1 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): got map 11 2022-03-29T17:50:00.961 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,749 [shard 0] osd - pg_epoch 11 pg[2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:00.961 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,749 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:00.961 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,749 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 NullEvt +create_info for pg: 2.4 2022-03-29T17:50:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=? pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:50:00,749 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.1 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): pg absent, did not create 2022-03-29T17:50:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.1 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): OSDState is active 2022-03-29T17:50:00.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,749 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:50:00.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 2} acting {1, 2} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:00.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:00.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:50:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:50:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 7.446034431s + 0.000000000s) 2022-03-29T17:50:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_advance_map(id=9, detail=PGAdvanceMap(pg=2.3 from=11 to=11 do_init)): start 2022-03-29T17:50:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=? pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): complete 2022-03-29T17:50:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.1 from 2 query_epoch 11 query: query(info 0'0 epoch_sent 11))): complete 2022-03-29T17:50:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001036 0 0.000000 2022-03-29T17:50:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Exiting state: Initial, entered at 1648576200.7494206, 0.0 spent on 0 events 2022-03-29T17:50:00.967 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:00.967 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:00.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:00.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000144 1 2022-03-29T17:50:00.750625+0000 2022-03-29T17:50:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Exiting state: Reset, entered at 1648576200.7505019, 1648576200.7506256 spent on 1 events 2022-03-29T17:50:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Entering state: Started 2022-03-29T17:50:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:00.970 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Entering state: Start 2022-03-29T17:50:00.970 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:50:00.970 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000039 0 0.000000 2022-03-29T17:50:00.970 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Exiting state: Start, entered at 1648576200.7507076, 0.0 spent on 0 events 2022-03-29T17:50:00.970 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:50:00.971 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:50:00.971 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:50:00.971 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:50:00.971 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.972 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,750 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:50:00.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:50:00.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:50:00.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering up_thru 8 < same_since 11, must notify monitor 2022-03-29T17:50:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:50:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:50:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,750 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:50:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:1 2022-03-29T17:50:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - Created 2.3 2022-03-29T17:50:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,751 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:50:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,751 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): pg present 2022-03-29T17:50:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.4 2022-03-29T17:50:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:50:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:50:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 NullEvt +create_info for pg: 2.3 2022-03-29T17:50:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:00.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.979 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,751 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e11 {2.3=11,2022-03-29T17:50:00.701078+0000,2.4=11,2022-03-29T17:50:00.701078+0000,2.5=11,2022-03-29T17:50:00.701078+0000,2.6=11,2022-03-29T17:50:00.701078+0000,2.7=11,2022-03-29T17:50:00.701078+0000}) v2): sub events complete 2022-03-29T17:50:00.979 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,751 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e11 {2.3=11,2022-03-29T17:50:00.701078+0000,2.4=11,2022-03-29T17:50:00.701078+0000,2.5=11,2022-03-29T17:50:00.701078+0000,2.6=11,2022-03-29T17:50:00.701078+0000,2.7=11,2022-03-29T17:50:00.701078+0000}) v2): complete 2022-03-29T17:50:00.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,752 [shard 0] osd - pg_advance_map(id=8, detail=PGAdvanceMap(pg=2.4 from=11 to=11 do_init)): complete 2022-03-29T17:50:00.980 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:00.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,755 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.980 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.980 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:00.980 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:00.981 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,755 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.981 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.981 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,755 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.981 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.982 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,755 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:00.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=? pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): complete 2022-03-29T17:50:00.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=? pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 NullEvt +create_info)): complete 2022-03-29T17:50:00.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - pg_advance_map(id=9, detail=PGAdvanceMap(pg=2.3 from=11 to=11 do_init)): complete 2022-03-29T17:50:00.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.3 2022-03-29T17:50:00.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:50:00.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - log is not dirty 2022-03-29T17:50:00.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:00.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:00.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,755 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:50:00.984 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,757 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:00.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,757 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:00.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,757 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:00.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,774 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:00.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,774 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.2 from 1 notify: (query:11 sent:11 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:50:00.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.2 from 1 notify: (query:11 sent:11 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 11 2022-03-29T17:50:00.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.2 from 1 notify: (query:11 sent:11 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:50:00.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:00.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.5 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): start 2022-03-29T17:50:00.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.2 from 1 notify: (query:11 sent:11 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 2.2 2022-03-29T17:50:00.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering got osd.1 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:00.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:00.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:00.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:50:00.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:50:00.996 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.058312 4 2022-03-29T17:50:00.721680+0000 2022-03-29T17:50:00.996 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,775 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576200.7173014, 1648576200.7216806 spent on 4 events 2022-03-29T17:50:00.996 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:50:00.996 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,775 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:50:00.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:00.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:00.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:00.997 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:50:00.997 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:00.998 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:50:00.998 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:00.998 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:50:00.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:50:00.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:50:00.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:50:00.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:50:01.000 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:50:01.000 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000332 0 0.000000 2022-03-29T17:50:01.000 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576200.7756882, 0.0 spent on 0 events 2022-03-29T17:50:01.000 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.001 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.001 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:50:01.001 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000052 0 0.000000 2022-03-29T17:50:01.001 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576200.7760594, 0.0 spent on 0 events 2022-03-29T17:50:01.002 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.002 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.2 from 1 notify: (query:11 sent:11 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:50:01.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.5 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): got map 11 2022-03-29T17:50:01.004 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:50:00,776 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.5 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): pg absent, did not create 2022-03-29T17:50:01.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.5 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): OSDState is active 2022-03-29T17:50:01.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:01.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.5 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): complete 2022-03-29T17:50:01.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:01.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.6 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): start 2022-03-29T17:50:01.011 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,776 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:01.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:01.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,776 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.2 from 1 notify: (query:11 sent:11 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:50:01.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.6 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): got map 11 2022-03-29T17:50:01.013 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:50:00,777 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.6 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): pg absent, did not create 2022-03-29T17:50:01.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.6 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): OSDState is active 2022-03-29T17:50:01.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:01.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.6 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): complete 2022-03-29T17:50:01.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - replicated_request(id=51, detail=RepRequest(from=1 req=osd_repop(client.4125.0:77 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'51, mlcod=11'51) v3)) start 2022-03-29T17:50:01.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - pg_epoch 11 pg[1.0( v 10'50 (0'0,10'50] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 mlcod 10'50 active append_log log((0'0,10'50], crt=10'50) {11'51 (10'50) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:77 2022-03-29T17:50:00.734860+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=10'50 lcod 10'50 mlcod 10'50 active add_log_entry 11'51 (10'50) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:77 2022-03-29T17:50:00.734860+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'51 lcod 10'50 mlcod 10'50 active append_log approx pg log length = 51 2022-03-29T17:50:01.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'51 lcod 10'50 mlcod 10'50 active append_log transaction_applied = 1 2022-03-29T17:50:01.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,777 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'51, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:01.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.7 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): start 2022-03-29T17:50:01.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.7 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): got map 11 2022-03-29T17:50:01.019 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:50:00,778 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.7 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): pg absent, did not create 2022-03-29T17:50:01.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.7 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): OSDState is active 2022-03-29T17:50:01.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:01.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:01.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.4 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): start 2022-03-29T17:50:01.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,778 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.7 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): complete 2022-03-29T17:50:01.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,779 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.4 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): got map 11 2022-03-29T17:50:01.022 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:50:00,779 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.4 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): pg absent, did not create 2022-03-29T17:50:01.023 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,779 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.4 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): OSDState is active 2022-03-29T17:50:01.023 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,779 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:01.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,779 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.4 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): complete 2022-03-29T17:50:01.039 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,782 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:01.040 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,782 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.0 from 1 notify: (query:11 sent:11 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:50:01.040 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.0 from 1 notify: (query:11 sent:11 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 11 2022-03-29T17:50:01.040 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.0 from 1 notify: (query:11 sent:11 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:50:01.041 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.0 from 1 notify: (query:11 sent:11 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 2.0 2022-03-29T17:50:01.041 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering got osd.1 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.041 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.041 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.041 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.042 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.042 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.066493 4 2022-03-29T17:50:00.724308+0000 2022-03-29T17:50:01.042 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576200.716929, 1648576200.7243083 spent on 4 events 2022-03-29T17:50:01.042 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:50:01.043 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:50:01.043 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.043 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.043 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:50:01.044 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.044 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:50:01.044 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.044 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:50:01.045 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.1 (up) accepted 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.045 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:50:01.045 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:50:01.045 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 1} async_recovery= 2022-03-29T17:50:01.046 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:50:01.046 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:50:01.046 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:50:01.046 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000352 0 0.000000 2022-03-29T17:50:01.046 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576200.7834718, 0.0 spent on 0 events 2022-03-29T17:50:01.047 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.047 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.047 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:50:01.047 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000065 0 0.000000 2022-03-29T17:50:01.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576200.7838595, 0.0 spent on 0 events 2022-03-29T17:50:01.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,783 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.048 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.049 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,783 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.0 from 1 notify: (query:11 sent:11 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:50:01.049 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,784 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.049 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,784 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:01.049 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,784 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.1 from 1 notify: (query:11 sent:11 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:50:01.050 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,784 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:01.050 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,784 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:01.050 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,784 [shard 0] osd - peering_event(id=5, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.0 from 1 notify: (query:11 sent:11 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:50:01.050 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,784 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.1 from 1 notify: (query:11 sent:11 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 11 2022-03-29T17:50:01.051 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,784 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.1 from 1 notify: (query:11 sent:11 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:50:01.051 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:01.051 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.3 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): start 2022-03-29T17:50:01.051 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.1 from 1 notify: (query:11 sent:11 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 2.1 2022-03-29T17:50:01.052 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering got osd.1 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.052 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.052 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.052 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.053 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.053 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.062971 4 2022-03-29T17:50:00.725704+0000 2022-03-29T17:50:01.053 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576200.7222614, 1648576200.725704 spent on 4 events 2022-03-29T17:50:01.053 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:50:01.054 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:50:01.054 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.054 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.054 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:50:01.055 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.055 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:50:01.055 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.055 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:50:01.056 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.1 (up) accepted 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.056 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:50:01.056 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:50:01.056 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 1} async_recovery= 2022-03-29T17:50:01.056 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:50:01.057 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:50:01.057 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:50:01.057 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000279 0 0.000000 2022-03-29T17:50:01.057 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576200.785297, 0.0 spent on 0 events 2022-03-29T17:50:01.058 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.058 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.058 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:50:01.058 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000046 0 0.000000 2022-03-29T17:50:01.059 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576200.785612, 0.0 spent on 0 events 2022-03-29T17:50:01.059 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.059 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,785 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.059 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.060 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.1 from 1 notify: (query:11 sent:11 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:50:01.060 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.060 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,785 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.3 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): got map 11 2022-03-29T17:50:01.060 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:50:00,785 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.3 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): pg absent, did not create 2022-03-29T17:50:01.061 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,786 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.3 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): OSDState is active 2022-03-29T17:50:01.061 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,786 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:01.061 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,786 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:01.061 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,786 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:01.061 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,786 [shard 0] osd - peering_event(id=7, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MQuery 2.3 from 1 query_epoch 11 query: query(info 0'0 epoch_sent 11))): complete 2022-03-29T17:50:01.062 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:00,786 [shard 0] osd - peering_event(id=6, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.1 from 1 notify: (query:11 sent:11 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:50:01.062 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,794 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@52685 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=3, pgs=11, cs=0, client_cookie=12990204181074666112, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:01.062 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:00,794 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:50:01.063 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,797 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@54730 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=3, pgs=12, cs=0, client_cookie=15312318518825473467, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:01.063 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:00,797 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:50:01.098 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,810 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@51139 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=4, pgs=13, cs=0, client_cookie=10183302347089209809, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:01.098 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,810 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:50:01.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,815 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:01.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,815 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.5 from 0 notify: (query:11 sent:11 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:50:01.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.5 from 0 notify: (query:11 sent:11 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 11 2022-03-29T17:50:01.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.5 from 0 notify: (query:11 sent:11 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:50:01.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.5 from 0 notify: (query:11 sent:11 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 2.5 2022-03-29T17:50:01.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering got osd.0 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.101 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.078927 4 2022-03-29T17:50:00.740513+0000 2022-03-29T17:50:01.101 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,816 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576200.7378237, 1648576200.740513 spent on 4 events 2022-03-29T17:50:01.101 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:50:01.101 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,816 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:50:01.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:50:01.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,816 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.103 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:50:01.103 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.103 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:50:01.103 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.104 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:50:01.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:50:01.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:50:01.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:50:01.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:50:01.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:50:01.105 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000352 0 0.000000 2022-03-29T17:50:01.105 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576200.8167942, 0.0 spent on 0 events 2022-03-29T17:50:01.105 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.106 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:50:01.106 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000046 0 0.000000 2022-03-29T17:50:01.106 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576200.8171785, 0.0 spent on 0 events 2022-03-29T17:50:01.107 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.107 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.5 from 0 notify: (query:11 sent:11 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:50:01.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:01.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.6 from 0 notify: (query:11 sent:11 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:50:01.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.6 from 0 notify: (query:11 sent:11 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 11 2022-03-29T17:50:01.109 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,817 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:01.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:01.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,817 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.5 from 0 notify: (query:11 sent:11 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:50:01.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.6 from 0 notify: (query:11 sent:11 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:50:01.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.6 from 0 notify: (query:11 sent:11 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 2.6 2022-03-29T17:50:01.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering got osd.0 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.077742 4 2022-03-29T17:50:00.746464+0000 2022-03-29T17:50:01.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576200.740581, 1648576200.7464643 spent on 4 events 2022-03-29T17:50:01.112 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:50:01.112 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:50:01.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:50:01.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.113 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:50:01.113 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.114 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:50:01.114 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.114 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:50:01.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:50:01.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:50:01.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:50:01.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:50:01.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:50:01.115 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000250 0 0.000000 2022-03-29T17:50:01.116 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576200.8183727, 0.0 spent on 0 events 2022-03-29T17:50:01.116 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.116 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:50:01.117 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000059 0 0.000000 2022-03-29T17:50:01.117 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576200.8186545, 0.0 spent on 0 events 2022-03-29T17:50:01.117 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.117 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,818 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.6 from 0 notify: (query:11 sent:11 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:50:01.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,818 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:01.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.7 from 0 notify: (query:11 sent:11 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:50:01.119 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,819 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:01.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:01.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.6 from 0 notify: (query:11 sent:11 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:50:01.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.7 from 0 notify: (query:11 sent:11 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 11 2022-03-29T17:50:01.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.7 from 0 notify: (query:11 sent:11 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:50:01.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.7 from 0 notify: (query:11 sent:11 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 2.7 2022-03-29T17:50:01.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering got osd.0 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.121 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.075215 4 2022-03-29T17:50:00.749230+0000 2022-03-29T17:50:01.122 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,819 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576200.7446055, 1648576200.7492301 spent on 4 events 2022-03-29T17:50:01.122 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:50:01.122 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,819 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:50:01.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,819 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:50:01.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.123 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:50:01.124 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.124 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:50:01.124 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.124 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:50:01.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:50:01.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:50:01.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:50:01.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:50:01.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:50:01.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000268 0 0.000000 2022-03-29T17:50:01.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,820 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576200.819868, 0.0 spent on 0 events 2022-03-29T17:50:01.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.127 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,820 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:50:01.127 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000044 0 0.000000 2022-03-29T17:50:01.127 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,820 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576200.8201694, 0.0 spent on 0 events 2022-03-29T17:50:01.128 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,820 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.128 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,820 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.7 from 0 notify: (query:11 sent:11 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:50:01.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:77 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - OpSequencer::finish_op_in_order: op=77, last_started=77, last_unblocked=77, last_completed=76 2022-03-29T17:50:01.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,820 [shard 0] osd - client_request(id=77, detail=m=[osd_op(client.4125.0:77 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:01.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.4 from 0 notify: (query:11 sent:11 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:50:01.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.4 from 0 notify: (query:11 sent:11 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 11 2022-03-29T17:50:01.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.4 from 0 notify: (query:11 sent:11 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:50:01.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.4 from 0 notify: (query:11 sent:11 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 2.4 2022-03-29T17:50:01.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering got osd.0 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.132 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.074373 4 2022-03-29T17:50:00.751951+0000 2022-03-29T17:50:01.132 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,821 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576200.7474082, 1648576200.7519512 spent on 4 events 2022-03-29T17:50:01.133 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:50:01.133 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,821 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:50:01.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:50:01.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,821 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.134 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:50:01.134 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.134 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:50:01.135 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.135 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:50:01.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:50:01.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:50:01.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:50:01.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:50:01.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:50:01.136 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000277 0 0.000000 2022-03-29T17:50:01.137 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576200.8218207, 0.0 spent on 0 events 2022-03-29T17:50:01.137 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.137 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:50:01.138 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000043 0 0.000000 2022-03-29T17:50:01.138 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576200.8221493, 0.0 spent on 0 events 2022-03-29T17:50:01.138 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.138 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.4 from 0 notify: (query:11 sent:11 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:50:01.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.139 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,822 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:01.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:01.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - client_request(id=78, detail=m=[osd_op(client.4125.0:78 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - client_request(id=78, detail=m=[osd_op(client.4125.0:78 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.7 from 0 notify: (query:11 sent:11 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:50:01.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - client_request(id=78, detail=m=[osd_op(client.4125.0:78 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - OpSequencer::start_op: op=78, last_started=77, last_unblocked=77, last_completed=77 2022-03-29T17:50:01.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - client_request(id=78, detail=m=[osd_op(client.4125.0:78 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,822 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:01.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:50:01.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:01.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:78 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,823 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,824 [shard 0] osd - OpSequencer::finish_op_in_order: op=78, last_started=78, last_unblocked=78, last_completed=77 2022-03-29T17:50:01.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,824 [shard 0] osd - client_request(id=78, detail=m=[osd_op(client.4125.0:78 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,824 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:01.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,824 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.3 from 2 notify: (query:11 sent:11 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:50:01.146 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,824 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:01.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,824 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:01.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,824 [shard 0] osd - client_request(id=79, detail=m=[osd_op(client.4125.0:79 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,824 [shard 0] osd - client_request(id=79, detail=m=[osd_op(client.4125.0:79 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.4 from 0 notify: (query:11 sent:11 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:50:01.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.3 from 2 notify: (query:11 sent:11 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 11 2022-03-29T17:50:01.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.3 from 2 notify: (query:11 sent:11 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:50:01.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - client_request(id=79, detail=m=[osd_op(client.4125.0:79 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - OpSequencer::start_op: op=79, last_started=78, last_unblocked=78, last_completed=78 2022-03-29T17:50:01.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.3 from 2 notify: (query:11 sent:11 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 2.3 2022-03-29T17:50:01.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering got osd.2 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:50:01.150 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.074835 4 2022-03-29T17:50:00.755879+0000 2022-03-29T17:50:01.150 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,825 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576200.750822, 1648576200.7558796 spent on 4 events 2022-03-29T17:50:01.150 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:50:01.151 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,825 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:50:01.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.152 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:50:01.152 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) 2022-03-29T17:50:01.152 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:50:01.152 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.2 (up) accepted 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.152 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:50:01.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:50:01.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 2} async_recovery= 2022-03-29T17:50:01.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:50:01.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:50:01.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:50:01.154 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,825 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000270 0 0.000000 2022-03-29T17:50:01.154 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,825 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576200.825699, 0.0 spent on 0 events 2022-03-29T17:50:01.155 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,826 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.155 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,826 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:50:01.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:50:01.155 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,826 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000050 0 0.000000 2022-03-29T17:50:01.156 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,826 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576200.8260005, 0.0 spent on 0 events 2022-03-29T17:50:01.156 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,826 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.156 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,826 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:50:01.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.3 from 2 notify: (query:11 sent:11 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:50:01.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - client_request(id=79, detail=m=[osd_op(client.4125.0:79 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op read 2022-03-29T17:50:01.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - handling op read on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - read: data length: 1 2022-03-29T17:50:01.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:79 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.159 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,826 [shard 0] osd - send_alive want=11 up_thru_wanted=11 2022-03-29T17:50:01.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,826 [shard 0] osd - send_alive want=11 <= up_thru_wanted=11; skipping 2022-03-29T17:50:01.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 MNotifyRec 2.3 from 2 notify: (query:11 sent:11 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:50:01.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - OpSequencer::finish_op_in_order: op=79, last_started=79, last_unblocked=79, last_completed=78 2022-03-29T17:50:01.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - client_request(id=79, detail=m=[osd_op(client.4125.0:79 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - client_request(id=80, detail=m=[osd_op(client.4125.0:80 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - client_request(id=80, detail=m=[osd_op(client.4125.0:80 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - client_request(id=80, detail=m=[osd_op(client.4125.0:80 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - OpSequencer::start_op: op=80, last_started=79, last_unblocked=79, last_completed=79 2022-03-29T17:50:01.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - client_request(id=80, detail=m=[osd_op(client.4125.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:50:01.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,827 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - read: data length: 16 2022-03-29T17:50:01.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - OpSequencer::finish_op_in_order: op=80, last_started=80, last_unblocked=80, last_completed=79 2022-03-29T17:50:01.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - client_request(id=80, detail=m=[osd_op(client.4125.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - client_request(id=81, detail=m=[osd_op(client.4125.0:81 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - client_request(id=81, detail=m=[osd_op(client.4125.0:81 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - client_request(id=81, detail=m=[osd_op(client.4125.0:81 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - OpSequencer::start_op: op=81, last_started=80, last_unblocked=80, last_completed=80 2022-03-29T17:50:01.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - client_request(id=81, detail=m=[osd_op(client.4125.0:81 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:50:01.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,828 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:50:01.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - OpSequencer::finish_op_in_order: op=81, last_started=81, last_unblocked=81, last_completed=80 2022-03-29T17:50:01.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - client_request(id=81, detail=m=[osd_op(client.4125.0:81 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - client_request(id=82, detail=m=[osd_op(client.4125.0:82 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - client_request(id=82, detail=m=[osd_op(client.4125.0:82 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,829 [shard 0] osd - client_request(id=82, detail=m=[osd_op(client.4125.0:82 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - OpSequencer::start_op: op=82, last_started=81, last_unblocked=81, last_completed=81 2022-03-29T17:50:01.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - client_request(id=82, detail=m=[osd_op(client.4125.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'51 mlcod 11'51 active+clean calc_trim_to_aggressive limit = 11'51 2022-03-29T17:50:01.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'51 mlcod 11'51 active+clean append_log log((0'0,11'51], crt=11'51) {11'52 (10'49) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:82 2022-03-29T17:50:00.829498+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'51 lcod 11'51 mlcod 11'51 active+clean add_log_entry 11'52 (10'49) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:82 2022-03-29T17:50:00.829498+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'52 lcod 11'51 mlcod 11'51 active+clean append_log approx pg log length = 52 2022-03-29T17:50:01.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'52 lcod 11'51 mlcod 11'51 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'52, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,830 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,831 [shard 0] osd - client_request(id=83, detail=m=[osd_op(client.4125.0:83 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,831 [shard 0] osd - client_request(id=83, detail=m=[osd_op(client.4125.0:83 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,831 [shard 0] osd - client_request(id=83, detail=m=[osd_op(client.4125.0:83 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,831 [shard 0] osd - OpSequencer::start_op: op=83, last_started=82, last_unblocked=82, last_completed=81 2022-03-29T17:50:01.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,831 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,831 [shard 0] osd - client_request(id=83, detail=m=[osd_op(client.4125.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - client_request(id=84, detail=m=[osd_op(client.4125.0:84 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - client_request(id=84, detail=m=[osd_op(client.4125.0:84 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'52 lcod 11'51 mlcod 11'51 active+clean calc_trim_to_aggressive limit = 11'51 2022-03-29T17:50:01.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'52 lcod 11'51 mlcod 11'51 active+clean append_log log((0'0,11'52], crt=11'52) {11'53 (11'52) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:83 2022-03-29T17:50:00.829540+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'52 lcod 11'51 mlcod 11'51 active+clean add_log_entry 11'53 (11'52) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:83 2022-03-29T17:50:00.829540+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'53 lcod 11'51 mlcod 11'51 active+clean append_log approx pg log length = 53 2022-03-29T17:50:01.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'53 lcod 11'51 mlcod 11'51 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'53, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - client_request(id=84, detail=m=[osd_op(client.4125.0:84 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,832 [shard 0] osd - OpSequencer::start_op: op=84, last_started=83, last_unblocked=83, last_completed=81 2022-03-29T17:50:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - client_request(id=84, detail=m=[osd_op(client.4125.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'53 lcod 11'51 mlcod 11'51 active+clean calc_trim_to_aggressive limit = 11'51 2022-03-29T17:50:01.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'53 lcod 11'51 mlcod 11'51 active+clean append_log log((0'0,11'53], crt=11'53) {11'54 (11'53) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:84 2022-03-29T17:50:00.829578+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'53 lcod 11'51 mlcod 11'51 active+clean add_log_entry 11'54 (11'53) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:84 2022-03-29T17:50:00.829578+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'54 lcod 11'51 mlcod 11'51 active+clean append_log approx pg log length = 54 2022-03-29T17:50:01.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'54 lcod 11'51 mlcod 11'51 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,833 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'54, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,834 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,834 [shard 0] osd - client_request(id=85, detail=m=[osd_op(client.4125.0:85 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,834 [shard 0] osd - client_request(id=85, detail=m=[osd_op(client.4125.0:85 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.188 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,835 [shard 0] osd - replicated_request(id=52, detail=RepRequest(from=1 req=osd_repop(client.4125.0:82 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'52, mlcod=11'52) v3)) start 2022-03-29T17:50:01.188 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,835 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'51 (0'0,11'51] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'51 mlcod 11'51 active append_log log((0'0,11'51], crt=11'51) {11'52 (10'49) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:82 2022-03-29T17:50:00.829498+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.188 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,835 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'51 lcod 11'51 mlcod 11'51 active add_log_entry 11'52 (10'49) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:82 2022-03-29T17:50:00.829498+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.188 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,835 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'52 lcod 11'51 mlcod 11'51 active append_log approx pg log length = 52 2022-03-29T17:50:01.189 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,835 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'52 lcod 11'51 mlcod 11'51 active append_log transaction_applied = 1 2022-03-29T17:50:01.189 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,835 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.189 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,835 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'52, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.189 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,835 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.190 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,836 [shard 0] osd - replicated_request(id=53, detail=RepRequest(from=1 req=osd_repop(client.4125.0:83 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'53, mlcod=11'53) v3)) start 2022-03-29T17:50:01.190 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,836 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'52 (0'0,11'52] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'52 mlcod 11'52 active append_log log((0'0,11'52], crt=11'52) {11'53 (11'52) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:83 2022-03-29T17:50:00.829540+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.190 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,836 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'52 lcod 11'52 mlcod 11'52 active add_log_entry 11'53 (11'52) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:83 2022-03-29T17:50:00.829540+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.190 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,836 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'53 lcod 11'52 mlcod 11'52 active append_log approx pg log length = 53 2022-03-29T17:50:01.191 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,836 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'53 lcod 11'52 mlcod 11'52 active append_log transaction_applied = 1 2022-03-29T17:50:01.191 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,836 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.191 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,836 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'53, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.191 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,837 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.191 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,837 [shard 0] osd - replicated_request(id=54, detail=RepRequest(from=1 req=osd_repop(client.4125.0:84 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'54, mlcod=11'54) v3)) start 2022-03-29T17:50:01.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,837 [shard 0] osd - client_request(id=85, detail=m=[osd_op(client.4125.0:85 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,837 [shard 0] osd - OpSequencer::start_op: op=85, last_started=84, last_unblocked=84, last_completed=81 2022-03-29T17:50:01.193 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,838 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'53 (0'0,11'53] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'53 mlcod 11'53 active append_log log((0'0,11'53], crt=11'53) {11'54 (11'53) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:84 2022-03-29T17:50:00.829578+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.194 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,838 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'53 lcod 11'53 mlcod 11'53 active add_log_entry 11'54 (11'53) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:84 2022-03-29T17:50:00.829578+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.194 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,838 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'54 lcod 11'53 mlcod 11'53 active append_log approx pg log length = 54 2022-03-29T17:50:01.194 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,838 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'54 lcod 11'53 mlcod 11'53 active append_log transaction_applied = 1 2022-03-29T17:50:01.194 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,838 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.195 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,838 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'54, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.195 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,838 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,837 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,838 [shard 0] osd - client_request(id=85, detail=m=[osd_op(client.4125.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,839 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,839 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,839 [shard 0] osd - client_request(id=86, detail=m=[osd_op(client.4125.0:86 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,839 [shard 0] osd - client_request(id=86, detail=m=[osd_op(client.4125.0:86 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,839 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,839 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'54 lcod 11'51 mlcod 11'51 active+clean calc_trim_to_aggressive limit = 11'51 2022-03-29T17:50:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'54 lcod 11'51 mlcod 11'51 active+clean append_log log((0'0,11'54], crt=11'54) {11'55 (11'54) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:85 2022-03-29T17:50:00.829605+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'54 lcod 11'51 mlcod 11'51 active+clean add_log_entry 11'55 (11'54) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:85 2022-03-29T17:50:00.829605+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'55 lcod 11'51 mlcod 11'51 active+clean append_log approx pg log length = 55 2022-03-29T17:50:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'55 lcod 11'51 mlcod 11'51 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'55, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,840 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.206 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,841 [shard 0] osd - replicated_request(id=55, detail=RepRequest(from=1 req=osd_repop(client.4125.0:85 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'55, mlcod=11'55) v3)) start 2022-03-29T17:50:01.206 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,841 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'54 (0'0,11'54] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'54 mlcod 11'54 active append_log log((0'0,11'54], crt=11'54) {11'55 (11'54) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:85 2022-03-29T17:50:00.829605+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.206 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,841 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'54 lcod 11'54 mlcod 11'54 active add_log_entry 11'55 (11'54) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:85 2022-03-29T17:50:00.829605+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.206 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,841 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'55 lcod 11'54 mlcod 11'54 active append_log approx pg log length = 55 2022-03-29T17:50:01.207 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,841 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'55 lcod 11'54 mlcod 11'54 active append_log transaction_applied = 1 2022-03-29T17:50:01.207 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,841 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.207 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,841 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'55, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.207 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,841 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,842 [shard 0] osd - client_request(id=86, detail=m=[osd_op(client.4125.0:86 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,842 [shard 0] osd - OpSequencer::start_op: op=86, last_started=85, last_unblocked=85, last_completed=81 2022-03-29T17:50:01.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - client_request(id=87, detail=m=[osd_op(client.4125.0:87 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - client_request(id=87, detail=m=[osd_op(client.4125.0:87 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - client_request(id=86, detail=m=[osd_op(client.4125.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - client_request(id=87, detail=m=[osd_op(client.4125.0:87 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - OpSequencer::start_op: op=87, last_started=86, last_unblocked=86, last_completed=81 2022-03-29T17:50:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'55 lcod 11'51 mlcod 11'51 active+clean calc_trim_to_aggressive limit = 11'51 2022-03-29T17:50:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'55 lcod 11'51 mlcod 11'51 active+clean append_log log((0'0,11'55], crt=11'55) {11'56 (11'55) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:86 2022-03-29T17:50:00.829672+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'55 lcod 11'51 mlcod 11'51 active+clean add_log_entry 11'56 (11'55) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:86 2022-03-29T17:50:00.829672+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'56 lcod 11'51 mlcod 11'51 active+clean append_log approx pg log length = 56 2022-03-29T17:50:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'51 lua=0'0 crt=11'56 lcod 11'51 mlcod 11'51 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,843 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,844 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'56, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,844 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,844 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,844 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.219 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,845 [shard 0] osd - replicated_request(id=56, detail=RepRequest(from=1 req=osd_repop(client.4125.0:86 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'56, mlcod=11'56) v3)) start 2022-03-29T17:50:01.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,845 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'55 (0'0,11'55] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'55 mlcod 11'55 active append_log log((0'0,11'55], crt=11'55) {11'56 (11'55) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:86 2022-03-29T17:50:00.829672+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,845 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'55 lcod 11'55 mlcod 11'55 active add_log_entry 11'56 (11'55) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:86 2022-03-29T17:50:00.829672+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,845 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'56 lcod 11'55 mlcod 11'55 active append_log approx pg log length = 56 2022-03-29T17:50:01.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,845 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'56 lcod 11'55 mlcod 11'55 active append_log transaction_applied = 1 2022-03-29T17:50:01.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,845 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.221 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,845 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'56, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.221 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,845 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,846 [shard 0] osd - client_request(id=87, detail=m=[osd_op(client.4125.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,846 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,846 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,846 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,846 [shard 0] osd - client_request(id=88, detail=m=[osd_op(client.4125.0:88 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,846 [shard 0] osd - client_request(id=88, detail=m=[osd_op(client.4125.0:88 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - OpSequencer::finish_op_in_order: op=82, last_started=87, last_unblocked=87, last_completed=81 2022-03-29T17:50:01.226 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.226 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.226 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'52 lua=0'0 crt=11'56 lcod 11'52 mlcod 11'52 active+clean calc_trim_to_aggressive limit = 11'52 2022-03-29T17:50:01.226 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'52 lua=0'0 crt=11'56 lcod 11'52 mlcod 11'52 active+clean append_log log((0'0,11'56], crt=11'56) {11'57 (11'56) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:87 2022-03-29T17:50:00.829710+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'52 lua=0'0 crt=11'56 lcod 11'52 mlcod 11'52 active+clean add_log_entry 11'57 (11'56) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:87 2022-03-29T17:50:00.829710+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'52 lua=0'0 crt=11'57 lcod 11'52 mlcod 11'52 active+clean append_log approx pg log length = 57 2022-03-29T17:50:01.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'52 lua=0'0 crt=11'57 lcod 11'52 mlcod 11'52 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,847 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'57, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,848 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.229 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,849 [shard 0] osd - replicated_request(id=57, detail=RepRequest(from=1 req=osd_repop(client.4125.0:87 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'57, mlcod=11'57) v3)) start 2022-03-29T17:50:01.229 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,849 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'56 (0'0,11'56] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'56 mlcod 11'56 active append_log log((0'0,11'56], crt=11'56) {11'57 (11'56) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:87 2022-03-29T17:50:00.829710+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.229 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,849 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'56 lcod 11'56 mlcod 11'56 active add_log_entry 11'57 (11'56) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:87 2022-03-29T17:50:00.829710+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.230 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,849 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'57 lcod 11'56 mlcod 11'56 active append_log approx pg log length = 57 2022-03-29T17:50:01.230 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,849 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'57 lcod 11'56 mlcod 11'56 active append_log transaction_applied = 1 2022-03-29T17:50:01.230 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,849 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.230 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,849 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'57, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.231 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,849 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,848 [shard 0] osd - client_request(id=82, detail=m=[osd_op(client.4125.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,850 [shard 0] osd - client_request(id=88, detail=m=[osd_op(client.4125.0:88 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,850 [shard 0] osd - OpSequencer::start_op: op=88, last_started=87, last_unblocked=87, last_completed=82 2022-03-29T17:50:01.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,850 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,851 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,851 [shard 0] osd - client_request(id=88, detail=m=[osd_op(client.4125.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,851 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,851 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,851 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - OpSequencer::finish_op_in_order: op=83, last_started=88, last_unblocked=88, last_completed=82 2022-03-29T17:50:01.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - client_request(id=83, detail=m=[osd_op(client.4125.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'53 lua=0'0 crt=11'57 lcod 11'53 mlcod 11'53 active+clean calc_trim_to_aggressive limit = 11'53 2022-03-29T17:50:01.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'53 lua=0'0 crt=11'57 lcod 11'53 mlcod 11'53 active+clean append_log log((0'0,11'57], crt=11'57) {11'58 (11'57) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:88 2022-03-29T17:50:00.829737+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'53 lua=0'0 crt=11'57 lcod 11'53 mlcod 11'53 active+clean add_log_entry 11'58 (11'57) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:88 2022-03-29T17:50:00.829737+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'53 lua=0'0 crt=11'58 lcod 11'53 mlcod 11'53 active+clean append_log approx pg log length = 58 2022-03-29T17:50:01.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'53 lua=0'0 crt=11'58 lcod 11'53 mlcod 11'53 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'58, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,852 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.240 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,853 [shard 0] osd - replicated_request(id=58, detail=RepRequest(from=1 req=osd_repop(client.4125.0:88 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'58, mlcod=11'58) v3)) start 2022-03-29T17:50:01.240 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,853 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'57 (0'0,11'57] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'57 mlcod 11'57 active append_log log((0'0,11'57], crt=11'57) {11'58 (11'57) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:88 2022-03-29T17:50:00.829737+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.241 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,853 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'57 lcod 11'57 mlcod 11'57 active add_log_entry 11'58 (11'57) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:88 2022-03-29T17:50:00.829737+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.241 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,853 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'58 lcod 11'57 mlcod 11'57 active append_log approx pg log length = 58 2022-03-29T17:50:01.241 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,853 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'58 lcod 11'57 mlcod 11'57 active append_log transaction_applied = 1 2022-03-29T17:50:01.241 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,853 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.242 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,853 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'58, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.242 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,853 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,855 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,855 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,855 [shard 0] osd - OpSequencer::finish_op_in_order: op=84, last_started=88, last_unblocked=88, last_completed=83 2022-03-29T17:50:01.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,855 [shard 0] osd - client_request(id=84, detail=m=[osd_op(client.4125.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,856 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,856 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,856 [shard 0] osd - OpSequencer::finish_op_in_order: op=85, last_started=88, last_unblocked=88, last_completed=84 2022-03-29T17:50:01.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,856 [shard 0] osd - client_request(id=85, detail=m=[osd_op(client.4125.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,857 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,857 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,857 [shard 0] osd - OpSequencer::finish_op_in_order: op=86, last_started=88, last_unblocked=88, last_completed=85 2022-03-29T17:50:01.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,857 [shard 0] osd - client_request(id=86, detail=m=[osd_op(client.4125.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,857 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,857 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,857 [shard 0] osd - OpSequencer::finish_op_in_order: op=87, last_started=88, last_unblocked=88, last_completed=86 2022-03-29T17:50:01.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,857 [shard 0] osd - client_request(id=87, detail=m=[osd_op(client.4125.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,858 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,858 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,858 [shard 0] osd - OpSequencer::finish_op_in_order: op=88, last_started=88, last_unblocked=88, last_completed=87 2022-03-29T17:50:01.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,858 [shard 0] osd - client_request(id=88, detail=m=[osd_op(client.4125.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,887 [shard 0] osd - client_request(id=89, detail=m=[osd_op(client.4125.0:89 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - client_request(id=89, detail=m=[osd_op(client.4125.0:89 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - client_request(id=89, detail=m=[osd_op(client.4125.0:89 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - OpSequencer::start_op: op=89, last_started=88, last_unblocked=88, last_completed=88 2022-03-29T17:50:01.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - client_request(id=89, detail=m=[osd_op(client.4125.0:89 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - client_request(id=90, detail=m=[osd_op(client.4125.0:90 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - client_request(id=90, detail=m=[osd_op(client.4125.0:90 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:01.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'58 mlcod 11'58 active+clean calc_trim_to_aggressive limit = 11'58 2022-03-29T17:50:01.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'58 mlcod 11'58 active+clean append_log log((0'0,11'58], crt=11'58) {11'59 (11'51) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:89 2022-03-29T17:50:00.887560+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'58 lua=0'0 crt=11'58 lcod 11'58 mlcod 11'58 active+clean add_log_entry 11'59 (11'51) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:89 2022-03-29T17:50:00.887560+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'58 lua=0'0 crt=11'59 lcod 11'58 mlcod 11'58 active+clean append_log approx pg log length = 59 2022-03-29T17:50:01.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'58 lua=0'0 crt=11'59 lcod 11'58 mlcod 11'58 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,888 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,889 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'59, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,889 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,889 [shard 0] osd - client_request(id=90, detail=m=[osd_op(client.4125.0:90 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.257 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,890 [shard 0] osd - replicated_request(id=59, detail=RepRequest(from=1 req=osd_repop(client.4125.0:89 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'59, mlcod=11'59) v3)) start 2022-03-29T17:50:01.257 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,890 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'58 (0'0,11'58] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'58 mlcod 11'58 active append_log log((0'0,11'58], crt=11'58) {11'59 (11'51) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:89 2022-03-29T17:50:00.887560+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,890 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'58 lcod 11'58 mlcod 11'58 active add_log_entry 11'59 (11'51) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:89 2022-03-29T17:50:00.887560+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,890 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'59 lcod 11'58 mlcod 11'58 active append_log approx pg log length = 59 2022-03-29T17:50:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,890 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'59 lcod 11'58 mlcod 11'58 active append_log transaction_applied = 1 2022-03-29T17:50:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,890 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,890 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'59, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.259 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,891 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,891 [shard 0] osd - OpSequencer::start_op: op=90, last_started=89, last_unblocked=89, last_completed=88 2022-03-29T17:50:01.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,891 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - client_request(id=90, detail=m=[osd_op(client.4125.0:90 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:01.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'58 lua=0'0 crt=11'59 lcod 11'58 mlcod 11'58 active+clean calc_trim_to_aggressive limit = 11'58 2022-03-29T17:50:01.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'58 lua=0'0 crt=11'59 lcod 11'58 mlcod 11'58 active+clean append_log log((0'0,11'59], crt=11'59) {11'60 (11'59) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:90 2022-03-29T17:50:00.887611+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'58 lua=0'0 crt=11'59 lcod 11'58 mlcod 11'58 active+clean add_log_entry 11'60 (11'59) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:90 2022-03-29T17:50:00.887611+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'58 lua=0'0 crt=11'60 lcod 11'58 mlcod 11'58 active+clean append_log approx pg log length = 60 2022-03-29T17:50:01.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'58 lua=0'0 crt=11'60 lcod 11'58 mlcod 11'58 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,892 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'60, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,893 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,894 [shard 0] osd - replicated_request(id=60, detail=RepRequest(from=1 req=osd_repop(client.4125.0:90 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'60, mlcod=11'60) v3)) start 2022-03-29T17:50:01.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,894 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'59 (0'0,11'59] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'59 mlcod 11'59 active append_log log((0'0,11'59], crt=11'59) {11'60 (11'59) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:90 2022-03-29T17:50:00.887611+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,894 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'59 lcod 11'59 mlcod 11'59 active add_log_entry 11'60 (11'59) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:90 2022-03-29T17:50:00.887611+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,894 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'60 lcod 11'59 mlcod 11'59 active append_log approx pg log length = 60 2022-03-29T17:50:01.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,894 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'60 lcod 11'59 mlcod 11'59 active append_log transaction_applied = 1 2022-03-29T17:50:01.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,894 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,894 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'60, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,894 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,895 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:89 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,896 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,896 [shard 0] osd - OpSequencer::finish_op_in_order: op=89, last_started=90, last_unblocked=90, last_completed=88 2022-03-29T17:50:01.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,896 [shard 0] osd - client_request(id=89, detail=m=[osd_op(client.4125.0:89 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,896 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:90 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,897 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,897 [shard 0] osd - OpSequencer::finish_op_in_order: op=90, last_started=90, last_unblocked=90, last_completed=89 2022-03-29T17:50:01.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,897 [shard 0] osd - client_request(id=90, detail=m=[osd_op(client.4125.0:90 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,937 [shard 0] osd - client_request(id=91, detail=m=[osd_op(client.4125.0:91 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,937 [shard 0] osd - client_request(id=91, detail=m=[osd_op(client.4125.0:91 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,937 [shard 0] osd - client_request(id=91, detail=m=[osd_op(client.4125.0:91 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,937 [shard 0] osd - OpSequencer::start_op: op=91, last_started=90, last_unblocked=90, last_completed=90 2022-03-29T17:50:01.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,937 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - client_request(id=91, detail=m=[osd_op(client.4125.0:91 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'60 mlcod 11'60 active+clean calc_trim_to_aggressive limit = 11'60 2022-03-29T17:50:01.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'60 mlcod 11'60 active+clean append_log log((0'0,11'60], crt=11'60) {11'61 (11'58) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:91 2022-03-29T17:50:00.937417+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'60 lua=0'0 crt=11'60 lcod 11'60 mlcod 11'60 active+clean add_log_entry 11'61 (11'58) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:91 2022-03-29T17:50:00.937417+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'60 lua=0'0 crt=11'61 lcod 11'60 mlcod 11'60 active+clean append_log approx pg log length = 61 2022-03-29T17:50:01.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'60 lua=0'0 crt=11'61 lcod 11'60 mlcod 11'60 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'61, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,938 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.278 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,939 [shard 0] osd - replicated_request(id=61, detail=RepRequest(from=1 req=osd_repop(client.4125.0:91 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'61, mlcod=11'61) v3)) start 2022-03-29T17:50:01.278 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,939 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'60 (0'0,11'60] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'60 mlcod 11'60 active append_log log((0'0,11'60], crt=11'60) {11'61 (11'58) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:91 2022-03-29T17:50:00.937417+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.278 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,940 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'60 lcod 11'60 mlcod 11'60 active add_log_entry 11'61 (11'58) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:91 2022-03-29T17:50:00.937417+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.278 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,940 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'61 lcod 11'60 mlcod 11'60 active append_log approx pg log length = 61 2022-03-29T17:50:01.279 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,940 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'61 lcod 11'60 mlcod 11'60 active append_log transaction_applied = 1 2022-03-29T17:50:01.279 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,940 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.279 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,940 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'61, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.279 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:00,940 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,980 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:91 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,981 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,981 [shard 0] osd - OpSequencer::finish_op_in_order: op=91, last_started=91, last_unblocked=91, last_completed=90 2022-03-29T17:50:01.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,981 [shard 0] osd - client_request(id=91, detail=m=[osd_op(client.4125.0:91 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,981 [shard 0] osd - client_request(id=92, detail=m=[osd_op(client.4125.0:92 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,981 [shard 0] osd - client_request(id=92, detail=m=[osd_op(client.4125.0:92 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,981 [shard 0] osd - client_request(id=92, detail=m=[osd_op(client.4125.0:92 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,981 [shard 0] osd - OpSequencer::start_op: op=92, last_started=91, last_unblocked=91, last_completed=91 2022-03-29T17:50:01.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - client_request(id=92, detail=m=[osd_op(client.4125.0:92 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:01.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:50:01.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:01.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.288 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,982 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:01.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:50:01.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,982 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.289 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:00,983 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:50:01.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'61 mlcod 11'61 active+clean calc_trim_to_aggressive limit = 11'61 2022-03-29T17:50:01.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'61 mlcod 11'61 active+clean append_log log((0'0,11'61], crt=11'61) {11'62 (11'60) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:92 2022-03-29T17:50:00.981467+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'61 lua=0'0 crt=11'61 lcod 11'61 mlcod 11'61 active+clean add_log_entry 11'62 (11'60) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:92 2022-03-29T17:50:00.981467+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'61 lua=0'0 crt=11'62 lcod 11'61 mlcod 11'61 active+clean append_log approx pg log length = 62 2022-03-29T17:50:01.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'61 lua=0'0 crt=11'62 lcod 11'61 mlcod 11'61 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'62, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:00,983 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.293 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,021 [shard 0] osd - replicated_request(id=62, detail=RepRequest(from=1 req=osd_repop(client.4125.0:92 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'62, mlcod=11'62) v3)) start 2022-03-29T17:50:01.293 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,021 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'61 (0'0,11'61] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'61 mlcod 11'61 active append_log log((0'0,11'61], crt=11'61) {11'62 (11'60) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:92 2022-03-29T17:50:00.981467+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.294 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,021 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'61 lcod 11'61 mlcod 11'61 active add_log_entry 11'62 (11'60) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:92 2022-03-29T17:50:00.981467+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.294 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,021 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'62 lcod 11'61 mlcod 11'61 active append_log approx pg log length = 62 2022-03-29T17:50:01.294 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,021 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'62 lcod 11'61 mlcod 11'61 active append_log transaction_applied = 1 2022-03-29T17:50:01.294 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,021 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.295 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,021 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'62, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.295 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,022 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,062 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:92 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,063 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,063 [shard 0] osd - OpSequencer::finish_op_in_order: op=92, last_started=92, last_unblocked=92, last_completed=91 2022-03-29T17:50:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,063 [shard 0] osd - client_request(id=92, detail=m=[osd_op(client.4125.0:92 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,068 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> unknown.? -@48770] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:48770/0 2022-03-29T17:50:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,069 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.? -@48770] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:50:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,069 [shard 0] monc - added challenge on [osd.1(client) v2:172.21.15.138:6800/35578 >> client.? -@48770] 2022-03-29T17:50:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,069 [shard 0] osd - handle_authentication mgr.x has caps allow * 2022-03-29T17:50:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,070 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.4137 172.21.15.138:0/1002772390@48770] established: gs=0, pgs=3, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,070 [shard 0] osd - client_request(id=93, detail=m=[osd_op(client.4137.0:1 1.0 1.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e11) v8]): start 2022-03-29T17:50:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,070 [shard 0] osd - client_request(id=93, detail=m=[osd_op(client.4137.0:1 1.0 1.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e11) v8]): in repeat 2022-03-29T17:50:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,070 [shard 0] osd - client_request(id=93, detail=m=[osd_op(client.4137.0:1 1.0 1.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e11) v8]) same_interval_since: 8 2022-03-29T17:50:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,070 [shard 0] osd - OpSequencer::start_op: op=93, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:50:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,070 [shard 0] osd - will be handling pg op pgnls 2022-03-29T17:50:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:50:01,070 [shard 0] osd - handling op pgnls 2022-03-29T17:50:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,071 [shard 0] osd - do_pgnls_common: finishing the 1st stage of pgls 2022-03-29T17:50:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,071 [shard 0] osd - do_pgnls_common: 1st done 2022-03-29T17:50:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,071 [shard 0] osd - operator(): response.entries.size()= 2022-03-29T17:50:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,071 [shard 0] osd - OpSequencer::finish_op_out_of_order: op=93, last_started=93, last_unblocked=93, last_completed=0 2022-03-29T17:50:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,071 [shard 0] osd - client_request(id=93, detail=m=[osd_op(client.4137.0:1 1.0 1:00000000::::head {pgnls start_epoch 11 in=39b out=49b} snapc 0={} ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e11) v8]): destroying 2022-03-29T17:50:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - client_request(id=94, detail=m=[osd_op(client.4125.0:93 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - client_request(id=94, detail=m=[osd_op(client.4125.0:93 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - client_request(id=94, detail=m=[osd_op(client.4125.0:93 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - OpSequencer::start_op: op=94, last_started=92, last_unblocked=92, last_completed=92 2022-03-29T17:50:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - client_request(id=94, detail=m=[osd_op(client.4125.0:93 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,072 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:50:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,073 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:50:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:50:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,073 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,073 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'62 mlcod 11'62 active+clean calc_trim_to_aggressive limit = 11'62 2022-03-29T17:50:01.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,074 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'62 mlcod 11'62 active+clean append_log log((0'0,11'62], crt=11'62) {11'63 (11'62) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:93 2022-03-29T17:50:01.072031+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,074 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'62 lua=0'0 crt=11'62 lcod 11'62 mlcod 11'62 active+clean add_log_entry 11'63 (11'62) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:93 2022-03-29T17:50:01.072031+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,074 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'62 lua=0'0 crt=11'63 lcod 11'62 mlcod 11'62 active+clean append_log approx pg log length = 63 2022-03-29T17:50:01.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,074 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'62 lua=0'0 crt=11'63 lcod 11'62 mlcod 11'62 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,074 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,074 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'63, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,074 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.314 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,107 [shard 0] osd - replicated_request(id=63, detail=RepRequest(from=1 req=osd_repop(client.4125.0:93 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'63, mlcod=11'63) v3)) start 2022-03-29T17:50:01.314 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,108 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'62 (0'0,11'62] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'62 mlcod 11'62 active append_log log((0'0,11'62], crt=11'62) {11'63 (11'62) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:93 2022-03-29T17:50:01.072031+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.314 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,108 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'62 lcod 11'62 mlcod 11'62 active add_log_entry 11'63 (11'62) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:93 2022-03-29T17:50:01.072031+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.315 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,108 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'63 lcod 11'62 mlcod 11'62 active append_log approx pg log length = 63 2022-03-29T17:50:01.315 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,108 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'63 lcod 11'62 mlcod 11'62 active append_log transaction_applied = 1 2022-03-29T17:50:01.315 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,108 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.315 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,108 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'63, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,108 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,149 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:93 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,150 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,150 [shard 0] osd - OpSequencer::finish_op_in_order: op=94, last_started=94, last_unblocked=94, last_completed=92 2022-03-29T17:50:01.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,150 [shard 0] osd - client_request(id=94, detail=m=[osd_op(client.4125.0:93 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,150 [shard 0] osd - client_request(id=95, detail=m=[osd_op(client.4125.0:94 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,150 [shard 0] osd - client_request(id=95, detail=m=[osd_op(client.4125.0:94 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,150 [shard 0] osd - client_request(id=95, detail=m=[osd_op(client.4125.0:94 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,150 [shard 0] osd - OpSequencer::start_op: op=95, last_started=94, last_unblocked=94, last_completed=94 2022-03-29T17:50:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - client_request(id=95, detail=m=[osd_op(client.4125.0:94 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,151 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:50:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:94 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - OpSequencer::finish_op_in_order: op=95, last_started=95, last_unblocked=95, last_completed=94 2022-03-29T17:50:01.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - client_request(id=95, detail=m=[osd_op(client.4125.0:94 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,152 [shard 0] osd - client_request(id=96, detail=m=[osd_op(client.4125.0:95 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - client_request(id=96, detail=m=[osd_op(client.4125.0:95 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - client_request(id=96, detail=m=[osd_op(client.4125.0:95 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - OpSequencer::start_op: op=96, last_started=95, last_unblocked=95, last_completed=95 2022-03-29T17:50:01.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - client_request(id=96, detail=m=[osd_op(client.4125.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op read 2022-03-29T17:50:01.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - handling op read on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - read: data length: 1 2022-03-29T17:50:01.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - OpSequencer::finish_op_in_order: op=96, last_started=96, last_unblocked=96, last_completed=95 2022-03-29T17:50:01.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,153 [shard 0] osd - client_request(id=96, detail=m=[osd_op(client.4125.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - client_request(id=97, detail=m=[osd_op(client.4125.0:96 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - client_request(id=97, detail=m=[osd_op(client.4125.0:96 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - client_request(id=97, detail=m=[osd_op(client.4125.0:96 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - OpSequencer::start_op: op=97, last_started=96, last_unblocked=96, last_completed=96 2022-03-29T17:50:01.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - client_request(id=97, detail=m=[osd_op(client.4125.0:96 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:50:01.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - read: data length: 16 2022-03-29T17:50:01.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,154 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:96 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - OpSequencer::finish_op_in_order: op=97, last_started=97, last_unblocked=97, last_completed=96 2022-03-29T17:50:01.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - client_request(id=97, detail=m=[osd_op(client.4125.0:96 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - client_request(id=98, detail=m=[osd_op(client.4125.0:97 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - client_request(id=98, detail=m=[osd_op(client.4125.0:97 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - client_request(id=98, detail=m=[osd_op(client.4125.0:97 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - OpSequencer::start_op: op=98, last_started=97, last_unblocked=97, last_completed=97 2022-03-29T17:50:01.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - client_request(id=98, detail=m=[osd_op(client.4125.0:97 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:50:01.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,155 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:50:01.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - OpSequencer::finish_op_in_order: op=98, last_started=98, last_unblocked=98, last_completed=97 2022-03-29T17:50:01.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - client_request(id=98, detail=m=[osd_op(client.4125.0:97 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - client_request(id=99, detail=m=[osd_op(client.4125.0:98 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - client_request(id=99, detail=m=[osd_op(client.4125.0:98 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - client_request(id=99, detail=m=[osd_op(client.4125.0:98 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - OpSequencer::start_op: op=99, last_started=98, last_unblocked=98, last_completed=98 2022-03-29T17:50:01.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - client_request(id=99, detail=m=[osd_op(client.4125.0:98 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:01.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,156 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:50:01.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:01.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.346 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,157 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:01.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:50:01.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.348 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,157 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:50:01.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,157 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,158 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'63 mlcod 11'63 active+clean calc_trim_to_aggressive limit = 11'63 2022-03-29T17:50:01.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,158 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'63 mlcod 11'63 active+clean append_log log((0'0,11'63], crt=11'63) {11'64 (11'63) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:98 2022-03-29T17:50:01.156329+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,158 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'63 lua=0'0 crt=11'63 lcod 11'63 mlcod 11'63 active+clean add_log_entry 11'64 (11'63) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:98 2022-03-29T17:50:01.156329+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,158 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'63 lua=0'0 crt=11'64 lcod 11'63 mlcod 11'63 active+clean append_log approx pg log length = 64 2022-03-29T17:50:01.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,158 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'63 lua=0'0 crt=11'64 lcod 11'63 mlcod 11'63 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,158 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,158 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'64, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,158 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.352 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,191 [shard 0] osd - replicated_request(id=64, detail=RepRequest(from=1 req=osd_repop(client.4125.0:98 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'64, mlcod=11'64) v3)) start 2022-03-29T17:50:01.352 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,192 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'63 (0'0,11'63] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'63 mlcod 11'63 active append_log log((0'0,11'63], crt=11'63) {11'64 (11'63) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:98 2022-03-29T17:50:01.156329+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.352 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,192 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'63 lcod 11'63 mlcod 11'63 active add_log_entry 11'64 (11'63) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:98 2022-03-29T17:50:01.156329+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.353 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,192 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'64 lcod 11'63 mlcod 11'63 active append_log approx pg log length = 64 2022-03-29T17:50:01.353 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,192 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'64 lcod 11'63 mlcod 11'63 active append_log transaction_applied = 1 2022-03-29T17:50:01.353 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,192 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.353 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,192 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'64, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.354 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,192 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.354 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,228 [shard 0] alienstore - stat 2022-03-29T17:50:01.354 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,228 [shard 0] alienstore - stat 2022-03-29T17:50:01.355 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,229 [shard 0] alienstore - stat 2022-03-29T17:50:01.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,232 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:98 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,233 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,233 [shard 0] osd - OpSequencer::finish_op_in_order: op=99, last_started=99, last_unblocked=99, last_completed=98 2022-03-29T17:50:01.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,233 [shard 0] osd - client_request(id=99, detail=m=[osd_op(client.4125.0:98 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.357 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,236 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> unknown.? -@36766] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:36766/0 2022-03-29T17:50:01.357 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,236 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> mgr.? -@36766] UPDATE: peer_type=mgr, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:50:01.358 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,236 [shard 0] monc - added challenge on [osd.0(client) v2:172.21.15.138:6805/35577 >> mgr.? -@36766] 2022-03-29T17:50:01.358 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,237 [shard 0] osd - handle_authentication mgr.x has caps allow * 2022-03-29T17:50:01.358 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,238 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> mgr.4102 172.21.15.138:0/35195@36766] established: gs=0, pgs=4, cs=0, client_cookie=16469120954575602451, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:01.359 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,238 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> unknown.? -@48782] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:48782/0 2022-03-29T17:50:01.359 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,239 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> mgr.? -@48782] UPDATE: peer_type=mgr, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:50:01.359 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,239 [shard 0] monc - added challenge on [osd.1(client) v2:172.21.15.138:6800/35578 >> mgr.? -@48782] 2022-03-29T17:50:01.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,239 [shard 0] osd - handle_authentication mgr.x has caps allow * 2022-03-29T17:50:01.360 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,240 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> mgr.4102 172.21.15.138:0/35195@48782] established: gs=0, pgs=5, cs=0, client_cookie=9260729540270013250, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,241 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> unknown.? -@41076] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:41076/0 2022-03-29T17:50:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,241 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? -@41076] UPDATE: peer_type=mgr, policy(lossy=true server=false standby=false resetcheck=false) 2022-03-29T17:50:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,241 [shard 0] monc - added challenge on [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? -@41076] 2022-03-29T17:50:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,242 [shard 0] osd - handle_authentication mgr.x has caps allow * 2022-03-29T17:50:01.362 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,242 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.4102 172.21.15.138:0/35195@41076] established: gs=0, pgs=6, cs=0, client_cookie=4745570671447675093, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:01.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - client_request(id=100, detail=m=[osd_op(client.4125.0:99 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - client_request(id=100, detail=m=[osd_op(client.4125.0:99 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - client_request(id=100, detail=m=[osd_op(client.4125.0:99 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - OpSequencer::start_op: op=100, last_started=99, last_unblocked=99, last_completed=99 2022-03-29T17:50:01.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - client_request(id=100, detail=m=[osd_op(client.4125.0:99 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:01.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,326 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:50:01.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.368 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,327 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:50:01.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:01.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:50:01.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:01.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.370 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,327 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,327 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'64 mlcod 11'64 active+clean calc_trim_to_aggressive limit = 11'64 2022-03-29T17:50:01.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'64 mlcod 11'64 active+clean append_log log((0'0,11'64], crt=11'64) {11'65 (11'64) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:99 2022-03-29T17:50:01.326004+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'64 lua=0'0 crt=11'64 lcod 11'64 mlcod 11'64 active+clean add_log_entry 11'65 (11'64) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:99 2022-03-29T17:50:01.326004+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'64 lua=0'0 crt=11'65 lcod 11'64 mlcod 11'64 active+clean append_log approx pg log length = 65 2022-03-29T17:50:01.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'64 lua=0'0 crt=11'65 lcod 11'64 mlcod 11'64 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'65, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,328 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,329 [shard 0] osd - replicated_request(id=65, detail=RepRequest(from=1 req=osd_repop(client.4125.0:99 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'65, mlcod=11'65) v3)) start 2022-03-29T17:50:01.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,329 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'64 (0'0,11'64] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'64 mlcod 11'64 active append_log log((0'0,11'64], crt=11'64) {11'65 (11'64) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:99 2022-03-29T17:50:01.326004+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.375 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,329 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'64 lcod 11'64 mlcod 11'64 active add_log_entry 11'65 (11'64) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:99 2022-03-29T17:50:01.326004+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.375 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,329 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'65 lcod 11'64 mlcod 11'64 active append_log approx pg log length = 65 2022-03-29T17:50:01.375 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,329 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'65 lcod 11'64 mlcod 11'64 active append_log transaction_applied = 1 2022-03-29T17:50:01.375 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,329 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.376 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,329 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'65, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.376 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,330 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,370 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:99 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - OpSequencer::finish_op_in_order: op=100, last_started=100, last_unblocked=100, last_completed=99 2022-03-29T17:50:01.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - client_request(id=100, detail=m=[osd_op(client.4125.0:99 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - client_request(id=101, detail=m=[osd_op(client.4125.0:100 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - client_request(id=101, detail=m=[osd_op(client.4125.0:100 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - client_request(id=101, detail=m=[osd_op(client.4125.0:100 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - OpSequencer::start_op: op=101, last_started=100, last_unblocked=100, last_completed=100 2022-03-29T17:50:01.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - client_request(id=101, detail=m=[osd_op(client.4125.0:100 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,371 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:01.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:50:01.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:01.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:100 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - OpSequencer::finish_op_in_order: op=101, last_started=101, last_unblocked=101, last_completed=100 2022-03-29T17:50:01.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,372 [shard 0] osd - client_request(id=101, detail=m=[osd_op(client.4125.0:100 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - client_request(id=102, detail=m=[osd_op(client.4125.0:101 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - client_request(id=102, detail=m=[osd_op(client.4125.0:101 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - client_request(id=102, detail=m=[osd_op(client.4125.0:101 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - OpSequencer::start_op: op=102, last_started=101, last_unblocked=101, last_completed=101 2022-03-29T17:50:01.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - client_request(id=102, detail=m=[osd_op(client.4125.0:101 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op read 2022-03-29T17:50:01.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - handling op read on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - read: data length: 1 2022-03-29T17:50:01.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:101 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,373 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - OpSequencer::finish_op_in_order: op=102, last_started=102, last_unblocked=102, last_completed=101 2022-03-29T17:50:01.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - client_request(id=102, detail=m=[osd_op(client.4125.0:101 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - client_request(id=103, detail=m=[osd_op(client.4125.0:102 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - client_request(id=103, detail=m=[osd_op(client.4125.0:102 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - client_request(id=103, detail=m=[osd_op(client.4125.0:102 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - OpSequencer::start_op: op=103, last_started=102, last_unblocked=102, last_completed=102 2022-03-29T17:50:01.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - client_request(id=103, detail=m=[osd_op(client.4125.0:102 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:50:01.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - read: data length: 16 2022-03-29T17:50:01.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:102 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,374 [shard 0] osd - OpSequencer::finish_op_in_order: op=103, last_started=103, last_unblocked=103, last_completed=102 2022-03-29T17:50:01.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - client_request(id=103, detail=m=[osd_op(client.4125.0:102 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - client_request(id=104, detail=m=[osd_op(client.4125.0:103 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - client_request(id=104, detail=m=[osd_op(client.4125.0:103 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - client_request(id=104, detail=m=[osd_op(client.4125.0:103 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - OpSequencer::start_op: op=104, last_started=103, last_unblocked=103, last_completed=103 2022-03-29T17:50:01.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - client_request(id=104, detail=m=[osd_op(client.4125.0:103 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:50:01.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:50:01.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,375 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:50:01.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,376 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:01.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,376 [shard 0] osd - OpSequencer::finish_op_in_order: op=104, last_started=104, last_unblocked=104, last_completed=103 2022-03-29T17:50:01.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,376 [shard 0] osd - client_request(id=104, detail=m=[osd_op(client.4125.0:103 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,376 [shard 0] osd - client_request(id=105, detail=m=[osd_op(client.4125.0:104 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,376 [shard 0] osd - client_request(id=105, detail=m=[osd_op(client.4125.0:104 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,376 [shard 0] osd - client_request(id=105, detail=m=[osd_op(client.4125.0:104 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,376 [shard 0] osd - OpSequencer::start_op: op=105, last_started=104, last_unblocked=104, last_completed=104 2022-03-29T17:50:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,376 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - client_request(id=105, detail=m=[osd_op(client.4125.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - client_request(id=106, detail=m=[osd_op(client.4125.0:105 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - client_request(id=106, detail=m=[osd_op(client.4125.0:105 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'65 mlcod 11'65 active+clean calc_trim_to_aggressive limit = 11'65 2022-03-29T17:50:01.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'65 mlcod 11'65 active+clean append_log log((0'0,11'65], crt=11'65) {11'66 (11'61) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:104 2022-03-29T17:50:01.376250+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'65 lcod 11'65 mlcod 11'65 active+clean add_log_entry 11'66 (11'61) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:104 2022-03-29T17:50:01.376250+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'66 lcod 11'65 mlcod 11'65 active+clean append_log approx pg log length = 66 2022-03-29T17:50:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'66 lcod 11'65 mlcod 11'65 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'66, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,377 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,378 [shard 0] osd - client_request(id=106, detail=m=[osd_op(client.4125.0:105 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,378 [shard 0] osd - OpSequencer::start_op: op=106, last_started=105, last_unblocked=105, last_completed=104 2022-03-29T17:50:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,378 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,378 [shard 0] osd - client_request(id=106, detail=m=[osd_op(client.4125.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,378 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,378 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'66 lcod 11'65 mlcod 11'65 active+clean calc_trim_to_aggressive limit = 11'65 2022-03-29T17:50:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'66 lcod 11'65 mlcod 11'65 active+clean append_log log((0'0,11'66], crt=11'66) {11'67 (11'66) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:105 2022-03-29T17:50:01.376280+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'66 lcod 11'65 mlcod 11'65 active+clean add_log_entry 11'67 (11'66) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:105 2022-03-29T17:50:01.376280+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'67 lcod 11'65 mlcod 11'65 active+clean append_log approx pg log length = 67 2022-03-29T17:50:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'67 lcod 11'65 mlcod 11'65 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'67, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,379 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - client_request(id=107, detail=m=[osd_op(client.4125.0:106 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - client_request(id=107, detail=m=[osd_op(client.4125.0:106 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - client_request(id=107, detail=m=[osd_op(client.4125.0:106 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - OpSequencer::start_op: op=107, last_started=106, last_unblocked=106, last_completed=104 2022-03-29T17:50:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - client_request(id=107, detail=m=[osd_op(client.4125.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'67 lcod 11'65 mlcod 11'65 active+clean calc_trim_to_aggressive limit = 11'65 2022-03-29T17:50:01.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'67 lcod 11'65 mlcod 11'65 active+clean append_log log((0'0,11'67], crt=11'67) {11'68 (11'67) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:106 2022-03-29T17:50:01.376301+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'67 lcod 11'65 mlcod 11'65 active+clean add_log_entry 11'68 (11'67) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:106 2022-03-29T17:50:01.376301+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'68 lcod 11'65 mlcod 11'65 active+clean append_log approx pg log length = 68 2022-03-29T17:50:01.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'68 lcod 11'65 mlcod 11'65 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,380 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'68, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,381 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,381 [shard 0] osd - client_request(id=108, detail=m=[osd_op(client.4125.0:107 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,381 [shard 0] osd - client_request(id=108, detail=m=[osd_op(client.4125.0:107 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - replicated_request(id=66, detail=RepRequest(from=1 req=osd_repop(client.4125.0:104 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'66, mlcod=11'66) v3)) start 2022-03-29T17:50:01.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'65 (0'0,11'65] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'65 mlcod 11'65 active append_log log((0'0,11'65], crt=11'65) {11'66 (11'61) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:104 2022-03-29T17:50:01.376250+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'65 lcod 11'65 mlcod 11'65 active add_log_entry 11'66 (11'61) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:104 2022-03-29T17:50:01.376250+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'66 lcod 11'65 mlcod 11'65 active append_log approx pg log length = 66 2022-03-29T17:50:01.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'66 lcod 11'65 mlcod 11'65 active append_log transaction_applied = 1 2022-03-29T17:50:01.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'66, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.424 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.424 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,382 [shard 0] osd - replicated_request(id=67, detail=RepRequest(from=1 req=osd_repop(client.4125.0:105 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'67, mlcod=11'67) v3)) start 2022-03-29T17:50:01.424 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,383 [shard 0] osd - replicated_request(id=68, detail=RepRequest(from=1 req=osd_repop(client.4125.0:106 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'68, mlcod=11'68) v3)) start 2022-03-29T17:50:01.424 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,384 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'66 (0'0,11'66] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'66 mlcod 11'66 active append_log log((0'0,11'66], crt=11'66) {11'67 (11'66) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:105 2022-03-29T17:50:01.376280+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.424 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,384 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'66 lcod 11'66 mlcod 11'66 active add_log_entry 11'67 (11'66) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:105 2022-03-29T17:50:01.376280+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,384 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'67 lcod 11'66 mlcod 11'66 active append_log approx pg log length = 67 2022-03-29T17:50:01.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,384 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'67 lcod 11'66 mlcod 11'66 active append_log transaction_applied = 1 2022-03-29T17:50:01.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,384 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,384 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'67, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.426 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,384 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,381 [shard 0] osd - client_request(id=108, detail=m=[osd_op(client.4125.0:107 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,384 [shard 0] osd - OpSequencer::start_op: op=108, last_started=107, last_unblocked=107, last_completed=104 2022-03-29T17:50:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - client_request(id=109, detail=m=[osd_op(client.4125.0:108 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - client_request(id=109, detail=m=[osd_op(client.4125.0:108 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - client_request(id=108, detail=m=[osd_op(client.4125.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - client_request(id=109, detail=m=[osd_op(client.4125.0:108 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - OpSequencer::start_op: op=109, last_started=108, last_unblocked=108, last_completed=104 2022-03-29T17:50:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'68 lcod 11'65 mlcod 11'65 active+clean calc_trim_to_aggressive limit = 11'65 2022-03-29T17:50:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'68 lcod 11'65 mlcod 11'65 active+clean append_log log((0'0,11'68], crt=11'68) {11'69 (11'68) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:107 2022-03-29T17:50:01.376323+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'68 lcod 11'65 mlcod 11'65 active+clean add_log_entry 11'69 (11'68) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:107 2022-03-29T17:50:01.376323+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'69 lcod 11'65 mlcod 11'65 active+clean append_log approx pg log length = 69 2022-03-29T17:50:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,385 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'69 lcod 11'65 mlcod 11'65 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'69, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.434 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'67 (0'0,11'67] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'67 mlcod 11'67 active append_log log((0'0,11'67], crt=11'67) {11'68 (11'67) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:106 2022-03-29T17:50:01.376301+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.434 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'67 lcod 11'67 mlcod 11'67 active add_log_entry 11'68 (11'67) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:106 2022-03-29T17:50:01.376301+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.434 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'68 lcod 11'67 mlcod 11'67 active append_log approx pg log length = 68 2022-03-29T17:50:01.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'68 lcod 11'67 mlcod 11'67 active append_log transaction_applied = 1 2022-03-29T17:50:01.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'68, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,386 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.437 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,388 [shard 0] osd - replicated_request(id=69, detail=RepRequest(from=1 req=osd_repop(client.4125.0:107 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'69, mlcod=11'69) v3)) start 2022-03-29T17:50:01.437 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,389 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'68 (0'0,11'68] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'68 mlcod 11'68 active append_log log((0'0,11'68], crt=11'68) {11'69 (11'68) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:107 2022-03-29T17:50:01.376323+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.438 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,389 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'68 lcod 11'68 mlcod 11'68 active add_log_entry 11'69 (11'68) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:107 2022-03-29T17:50:01.376323+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.438 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,389 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'69 lcod 11'68 mlcod 11'68 active append_log approx pg log length = 69 2022-03-29T17:50:01.438 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,389 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'69 lcod 11'68 mlcod 11'68 active append_log transaction_applied = 1 2022-03-29T17:50:01.438 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,389 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.438 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,389 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'69, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.439 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,389 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.441 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.441 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - client_request(id=109, detail=m=[osd_op(client.4125.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - client_request(id=110, detail=m=[osd_op(client.4125.0:109 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - client_request(id=110, detail=m=[osd_op(client.4125.0:109 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.443 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.443 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.443 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.443 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'69 lcod 11'65 mlcod 11'65 active+clean calc_trim_to_aggressive limit = 11'65 2022-03-29T17:50:01.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'69 lcod 11'65 mlcod 11'65 active+clean append_log log((0'0,11'69], crt=11'69) {11'70 (11'69) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:108 2022-03-29T17:50:01.376573+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'69 lcod 11'65 mlcod 11'65 active+clean add_log_entry 11'70 (11'69) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:108 2022-03-29T17:50:01.376573+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'70 lcod 11'65 mlcod 11'65 active+clean append_log approx pg log length = 70 2022-03-29T17:50:01.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'65 lua=0'0 crt=11'70 lcod 11'65 mlcod 11'65 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,390 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'70, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,391 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,391 [shard 0] osd - client_request(id=110, detail=m=[osd_op(client.4125.0:109 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,391 [shard 0] osd - OpSequencer::start_op: op=110, last_started=109, last_unblocked=109, last_completed=104 2022-03-29T17:50:01.447 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,392 [shard 0] osd - replicated_request(id=70, detail=RepRequest(from=1 req=osd_repop(client.4125.0:108 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'70, mlcod=11'70) v3)) start 2022-03-29T17:50:01.447 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,392 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'69 (0'0,11'69] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'69 mlcod 11'69 active append_log log((0'0,11'69], crt=11'69) {11'70 (11'69) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:108 2022-03-29T17:50:01.376573+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.447 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,392 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'69 lcod 11'69 mlcod 11'69 active add_log_entry 11'70 (11'69) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:108 2022-03-29T17:50:01.376573+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.447 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,392 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'70 lcod 11'69 mlcod 11'69 active append_log approx pg log length = 70 2022-03-29T17:50:01.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,392 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'70 lcod 11'69 mlcod 11'69 active append_log transaction_applied = 1 2022-03-29T17:50:01.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,392 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,392 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'70, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,393 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.451 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,393 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.451 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,394 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.451 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,394 [shard 0] osd - client_request(id=110, detail=m=[osd_op(client.4125.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.451 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,394 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,394 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'66 lua=0'0 crt=11'70 lcod 11'66 mlcod 11'66 active+clean calc_trim_to_aggressive limit = 11'66 2022-03-29T17:50:01.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'66 lua=0'0 crt=11'70 lcod 11'66 mlcod 11'66 active+clean append_log log((0'0,11'70], crt=11'70) {11'71 (11'70) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:109 2022-03-29T17:50:01.376621+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'66 lua=0'0 crt=11'70 lcod 11'66 mlcod 11'66 active+clean add_log_entry 11'71 (11'70) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:109 2022-03-29T17:50:01.376621+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'66 lua=0'0 crt=11'71 lcod 11'66 mlcod 11'66 active+clean append_log approx pg log length = 71 2022-03-29T17:50:01.454 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'66 lua=0'0 crt=11'71 lcod 11'66 mlcod 11'66 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.454 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.454 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'71, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.454 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.455 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,395 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.455 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,396 [shard 0] osd - OpSequencer::finish_op_in_order: op=105, last_started=110, last_unblocked=110, last_completed=104 2022-03-29T17:50:01.456 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,396 [shard 0] osd - replicated_request(id=71, detail=RepRequest(from=1 req=osd_repop(client.4125.0:109 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'71, mlcod=11'71) v3)) start 2022-03-29T17:50:01.456 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'70 (0'0,11'70] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'70 mlcod 11'70 active append_log log((0'0,11'70], crt=11'70) {11'71 (11'70) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:109 2022-03-29T17:50:01.376621+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.457 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'70 lcod 11'70 mlcod 11'70 active add_log_entry 11'71 (11'70) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:109 2022-03-29T17:50:01.376621+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.457 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'71 lcod 11'70 mlcod 11'70 active append_log approx pg log length = 71 2022-03-29T17:50:01.457 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'71 lcod 11'70 mlcod 11'70 active append_log transaction_applied = 1 2022-03-29T17:50:01.457 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.457 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'71, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.458 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.461 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,396 [shard 0] osd - client_request(id=105, detail=m=[osd_op(client.4125.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.461 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - client_request(id=111, detail=m=[osd_op(client.4125.0:110 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.462 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,397 [shard 0] osd - client_request(id=111, detail=m=[osd_op(client.4125.0:110 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.462 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,398 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.462 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,398 [shard 0] osd - client_request(id=111, detail=m=[osd_op(client.4125.0:110 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.462 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,398 [shard 0] osd - OpSequencer::start_op: op=111, last_started=110, last_unblocked=110, last_completed=105 2022-03-29T17:50:01.463 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,399 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.463 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,399 [shard 0] osd - OpSequencer::finish_op_in_order: op=106, last_started=111, last_unblocked=111, last_completed=105 2022-03-29T17:50:01.463 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,399 [shard 0] osd - client_request(id=106, detail=m=[osd_op(client.4125.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.463 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,399 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.463 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - client_request(id=112, detail=m=[osd_op(client.4125.0:111 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.464 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - client_request(id=112, detail=m=[osd_op(client.4125.0:111 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.464 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - client_request(id=111, detail=m=[osd_op(client.4125.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.464 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.464 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.465 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.465 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.465 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.465 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.465 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'67 lua=0'0 crt=11'71 lcod 11'67 mlcod 11'67 active+clean calc_trim_to_aggressive limit = 11'67 2022-03-29T17:50:01.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'67 lua=0'0 crt=11'71 lcod 11'67 mlcod 11'67 active+clean append_log log((0'0,11'71], crt=11'71) {11'72 (11'71) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:110 2022-03-29T17:50:01.376638+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'67 lua=0'0 crt=11'71 lcod 11'67 mlcod 11'67 active+clean add_log_entry 11'72 (11'71) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:110 2022-03-29T17:50:01.376638+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'67 lua=0'0 crt=11'72 lcod 11'67 mlcod 11'67 active+clean append_log approx pg log length = 72 2022-03-29T17:50:01.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'67 lua=0'0 crt=11'72 lcod 11'67 mlcod 11'67 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'72, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,400 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,401 [shard 0] osd - client_request(id=112, detail=m=[osd_op(client.4125.0:111 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,401 [shard 0] osd - OpSequencer::start_op: op=112, last_started=111, last_unblocked=111, last_completed=106 2022-03-29T17:50:01.468 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,401 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - replicated_request(id=72, detail=RepRequest(from=1 req=osd_repop(client.4125.0:110 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'72, mlcod=11'72) v3)) start 2022-03-29T17:50:01.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'71 (0'0,11'71] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'71 mlcod 11'71 active append_log log((0'0,11'71], crt=11'71) {11'72 (11'71) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:110 2022-03-29T17:50:01.376638+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.470 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'71 lcod 11'71 mlcod 11'71 active add_log_entry 11'72 (11'71) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:110 2022-03-29T17:50:01.376638+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.470 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'72 lcod 11'71 mlcod 11'71 active append_log approx pg log length = 72 2022-03-29T17:50:01.470 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'72 lcod 11'71 mlcod 11'71 active append_log transaction_applied = 1 2022-03-29T17:50:01.470 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'72, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.473 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,402 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.474 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,403 [shard 0] osd - OpSequencer::finish_op_in_order: op=107, last_started=112, last_unblocked=112, last_completed=106 2022-03-29T17:50:01.474 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,403 [shard 0] osd - client_request(id=107, detail=m=[osd_op(client.4125.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.474 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,403 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.474 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,403 [shard 0] osd - client_request(id=113, detail=m=[osd_op(client.4125.0:112 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,403 [shard 0] osd - client_request(id=113, detail=m=[osd_op(client.4125.0:112 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - client_request(id=112, detail=m=[osd_op(client.4125.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131600~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'68 lua=0'0 crt=11'72 lcod 11'68 mlcod 11'68 active+clean calc_trim_to_aggressive limit = 11'68 2022-03-29T17:50:01.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'68 lua=0'0 crt=11'72 lcod 11'68 mlcod 11'68 active+clean append_log log((0'0,11'72], crt=11'72) {11'73 (11'72) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:111 2022-03-29T17:50:01.376659+0000 0 ObjectCleanRegions clean_offsets: [0~131600,131604~18446744073709420011], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'68 lua=0'0 crt=11'72 lcod 11'68 mlcod 11'68 active+clean add_log_entry 11'73 (11'72) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:111 2022-03-29T17:50:01.376659+0000 0 ObjectCleanRegions clean_offsets: [0~131600,131604~18446744073709420011], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'68 lua=0'0 crt=11'73 lcod 11'68 mlcod 11'68 active+clean append_log approx pg log length = 73 2022-03-29T17:50:01.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'68 lua=0'0 crt=11'73 lcod 11'68 mlcod 11'68 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.478 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.478 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'73, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.478 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.478 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,404 [shard 0] osd - client_request(id=113, detail=m=[osd_op(client.4125.0:112 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,405 [shard 0] osd - OpSequencer::start_op: op=113, last_started=112, last_unblocked=112, last_completed=107 2022-03-29T17:50:01.480 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,405 [shard 0] osd - replicated_request(id=73, detail=RepRequest(from=1 req=osd_repop(client.4125.0:111 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'73, mlcod=11'73) v3)) start 2022-03-29T17:50:01.480 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,405 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'72 (0'0,11'72] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'72 mlcod 11'72 active append_log log((0'0,11'72], crt=11'72) {11'73 (11'72) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:111 2022-03-29T17:50:01.376659+0000 0 ObjectCleanRegions clean_offsets: [0~131600,131604~18446744073709420011], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.480 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,405 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'72 lcod 11'72 mlcod 11'72 active add_log_entry 11'73 (11'72) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:111 2022-03-29T17:50:01.376659+0000 0 ObjectCleanRegions clean_offsets: [0~131600,131604~18446744073709420011], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,405 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'73 lcod 11'72 mlcod 11'72 active append_log approx pg log length = 73 2022-03-29T17:50:01.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,405 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'73 lcod 11'72 mlcod 11'72 active append_log transaction_applied = 1 2022-03-29T17:50:01.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,405 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,405 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'73, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.482 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,406 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,406 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,407 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,407 [shard 0] osd - client_request(id=113, detail=m=[osd_op(client.4125.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131604~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,407 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,407 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,407 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,407 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,407 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,407 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'69 lua=0'0 crt=11'73 lcod 11'69 mlcod 11'69 active+clean calc_trim_to_aggressive limit = 11'69 2022-03-29T17:50:01.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'69 lua=0'0 crt=11'73 lcod 11'69 mlcod 11'69 active+clean append_log log((0'0,11'73], crt=11'73) {11'74 (11'73) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:112 2022-03-29T17:50:01.376700+0000 0 ObjectCleanRegions clean_offsets: [0~131604,197140~18446744073709354475], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'69 lua=0'0 crt=11'73 lcod 11'69 mlcod 11'69 active+clean add_log_entry 11'74 (11'73) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:112 2022-03-29T17:50:01.376700+0000 0 ObjectCleanRegions clean_offsets: [0~131604,197140~18446744073709354475], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'69 lua=0'0 crt=11'74 lcod 11'69 mlcod 11'69 active+clean append_log approx pg log length = 74 2022-03-29T17:50:01.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'69 lua=0'0 crt=11'74 lcod 11'69 mlcod 11'69 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'74, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,408 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,409 [shard 0] osd - replicated_request(id=74, detail=RepRequest(from=1 req=osd_repop(client.4125.0:112 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'74, mlcod=11'74) v3)) start 2022-03-29T17:50:01.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,409 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'73 (0'0,11'73] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'73 mlcod 11'73 active append_log log((0'0,11'73], crt=11'73) {11'74 (11'73) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:112 2022-03-29T17:50:01.376700+0000 0 ObjectCleanRegions clean_offsets: [0~131604,197140~18446744073709354475], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,409 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'73 lcod 11'73 mlcod 11'73 active add_log_entry 11'74 (11'73) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:112 2022-03-29T17:50:01.376700+0000 0 ObjectCleanRegions clean_offsets: [0~131604,197140~18446744073709354475], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,409 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'74 lcod 11'73 mlcod 11'73 active append_log approx pg log length = 74 2022-03-29T17:50:01.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,409 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'74 lcod 11'73 mlcod 11'73 active append_log transaction_applied = 1 2022-03-29T17:50:01.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,409 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,409 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'74, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,409 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,410 [shard 0] osd - OpSequencer::finish_op_in_order: op=108, last_started=113, last_unblocked=113, last_completed=107 2022-03-29T17:50:01.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,410 [shard 0] osd - client_request(id=108, detail=m=[osd_op(client.4125.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,410 [shard 0] osd - client_request(id=114, detail=m=[osd_op(client.4125.0:113 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,410 [shard 0] osd - client_request(id=114, detail=m=[osd_op(client.4125.0:113 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,411 [shard 0] osd - client_request(id=114, detail=m=[osd_op(client.4125.0:113 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,411 [shard 0] osd - OpSequencer::start_op: op=114, last_started=113, last_unblocked=113, last_completed=108 2022-03-29T17:50:01.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,411 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,411 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - client_request(id=114, detail=m=[osd_op(client.4125.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197140~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - client_request(id=115, detail=m=[osd_op(client.4125.0:114 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - client_request(id=115, detail=m=[osd_op(client.4125.0:114 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - OpSequencer::finish_op_in_order: op=109, last_started=114, last_unblocked=114, last_completed=108 2022-03-29T17:50:01.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - client_request(id=115, detail=m=[osd_op(client.4125.0:114 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - OpSequencer::start_op: op=115, last_started=114, last_unblocked=114, last_completed=109 2022-03-29T17:50:01.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,412 [shard 0] osd - client_request(id=109, detail=m=[osd_op(client.4125.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'74 lcod 11'70 mlcod 11'70 active+clean calc_trim_to_aggressive limit = 11'70 2022-03-29T17:50:01.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'74 lcod 11'70 mlcod 11'70 active+clean append_log log((0'0,11'74], crt=11'74) {11'75 (11'74) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:113 2022-03-29T17:50:01.376716+0000 0 ObjectCleanRegions clean_offsets: [0~197140,197144~18446744073709354471], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'74 lcod 11'70 mlcod 11'70 active+clean add_log_entry 11'75 (11'74) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:113 2022-03-29T17:50:01.376716+0000 0 ObjectCleanRegions clean_offsets: [0~197140,197144~18446744073709354471], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'75 lcod 11'70 mlcod 11'70 active+clean append_log approx pg log length = 75 2022-03-29T17:50:01.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'75 lcod 11'70 mlcod 11'70 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'75, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,413 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.502 INFO:tasks.ceph.osd.1.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/include/inline_memory.h:112:9: runtime error: load of misaligned address 0x7fded9886816 for type 'uint32_t', which requires 4 byte alignment 2022-03-29T17:50:01.502 INFO:tasks.ceph.osd.1.smithi138.stderr:0x7fded9886816: note: pointer points here 2022-03-29T17:50:01.502 INFO:tasks.ceph.osd.1.smithi138.stderr: bb cf 01 3f bd b7 81 a8 0e 60 fa f8 26 ff ff ff ff 4f 3a 98 25 11 04 29 00 00 00 08 00 fd 00 00 2022-03-29T17:50:01.502 INFO:tasks.ceph.osd.1.smithi138.stderr: ^ 2022-03-29T17:50:01.503 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,414 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,414 [shard 0] osd - replicated_request(id=75, detail=RepRequest(from=1 req=osd_repop(client.4125.0:113 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'75, mlcod=11'75) v3)) start 2022-03-29T17:50:01.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,414 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'74 (0'0,11'74] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'74 mlcod 11'74 active append_log log((0'0,11'74], crt=11'74) {11'75 (11'74) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:113 2022-03-29T17:50:01.376716+0000 0 ObjectCleanRegions clean_offsets: [0~197140,197144~18446744073709354471], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,414 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'74 lcod 11'74 mlcod 11'74 active add_log_entry 11'75 (11'74) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:113 2022-03-29T17:50:01.376716+0000 0 ObjectCleanRegions clean_offsets: [0~197140,197144~18446744073709354471], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,414 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'75 lcod 11'74 mlcod 11'74 active append_log approx pg log length = 75 2022-03-29T17:50:01.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,414 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'75 lcod 11'74 mlcod 11'74 active append_log transaction_applied = 1 2022-03-29T17:50:01.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,414 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,414 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'75, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,415 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.508 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,415 [shard 0] osd - client_request(id=115, detail=m=[osd_op(client.4125.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197144~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.508 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,415 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.508 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,415 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,415 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,415 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,415 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'75 lcod 11'70 mlcod 11'70 active+clean calc_trim_to_aggressive limit = 11'70 2022-03-29T17:50:01.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'75 lcod 11'70 mlcod 11'70 active+clean append_log log((0'0,11'75], crt=11'75) {11'76 (11'75) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:114 2022-03-29T17:50:01.376790+0000 0 ObjectCleanRegions clean_offsets: [0~197144,197148~18446744073709354467], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'75 lcod 11'70 mlcod 11'70 active+clean add_log_entry 11'76 (11'75) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:114 2022-03-29T17:50:01.376790+0000 0 ObjectCleanRegions clean_offsets: [0~197144,197148~18446744073709354467], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'76 lcod 11'70 mlcod 11'70 active+clean append_log approx pg log length = 76 2022-03-29T17:50:01.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'70 lua=0'0 crt=11'76 lcod 11'70 mlcod 11'70 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'76, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,416 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - client_request(id=116, detail=m=[osd_op(client.4125.0:115 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - client_request(id=116, detail=m=[osd_op(client.4125.0:115 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - replicated_request(id=76, detail=RepRequest(from=1 req=osd_repop(client.4125.0:114 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'76, mlcod=11'76) v3)) start 2022-03-29T17:50:01.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'75 (0'0,11'75] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'75 mlcod 11'75 active append_log log((0'0,11'75], crt=11'75) {11'76 (11'75) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:114 2022-03-29T17:50:01.376790+0000 0 ObjectCleanRegions clean_offsets: [0~197144,197148~18446744073709354467], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'75 lcod 11'75 mlcod 11'75 active add_log_entry 11'76 (11'75) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:114 2022-03-29T17:50:01.376790+0000 0 ObjectCleanRegions clean_offsets: [0~197144,197148~18446744073709354467], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'76 lcod 11'75 mlcod 11'75 active append_log approx pg log length = 76 2022-03-29T17:50:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'76 lcod 11'75 mlcod 11'75 active append_log transaction_applied = 1 2022-03-29T17:50:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,417 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'76, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,418 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,418 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,418 [shard 0] osd - OpSequencer::finish_op_in_order: op=110, last_started=115, last_unblocked=115, last_completed=109 2022-03-29T17:50:01.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,418 [shard 0] osd - client_request(id=110, detail=m=[osd_op(client.4125.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,418 [shard 0] osd - client_request(id=116, detail=m=[osd_op(client.4125.0:115 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,418 [shard 0] osd - OpSequencer::start_op: op=116, last_started=115, last_unblocked=115, last_completed=110 2022-03-29T17:50:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,419 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,419 [shard 0] osd - client_request(id=116, detail=m=[osd_op(client.4125.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197148~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,419 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,419 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,419 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'72 lua=0'0 crt=11'76 lcod 11'72 mlcod 11'72 active+clean calc_trim_to_aggressive limit = 11'72 2022-03-29T17:50:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'72 lua=0'0 crt=11'76 lcod 11'72 mlcod 11'72 active+clean append_log log((0'0,11'76], crt=11'76) {11'77 (11'76) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:115 2022-03-29T17:50:01.376839+0000 0 ObjectCleanRegions clean_offsets: [0~197148,262684~18446744073709288931], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'72 lua=0'0 crt=11'76 lcod 11'72 mlcod 11'72 active+clean add_log_entry 11'77 (11'76) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:115 2022-03-29T17:50:01.376839+0000 0 ObjectCleanRegions clean_offsets: [0~197148,262684~18446744073709288931], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'72 lua=0'0 crt=11'77 lcod 11'72 mlcod 11'72 active+clean append_log approx pg log length = 77 2022-03-29T17:50:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,420 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'72 lua=0'0 crt=11'77 lcod 11'72 mlcod 11'72 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,421 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,421 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'77, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,421 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,421 [shard 0] osd - OpSequencer::finish_op_in_order: op=111, last_started=116, last_unblocked=116, last_completed=110 2022-03-29T17:50:01.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,421 [shard 0] osd - client_request(id=111, detail=m=[osd_op(client.4125.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,421 [shard 0] osd - client_request(id=117, detail=m=[osd_op(client.4125.0:116 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,422 [shard 0] osd - replicated_request(id=77, detail=RepRequest(from=1 req=osd_repop(client.4125.0:115 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'77, mlcod=11'77) v3)) start 2022-03-29T17:50:01.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,422 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'76 (0'0,11'76] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'76 mlcod 11'76 active append_log log((0'0,11'76], crt=11'76) {11'77 (11'76) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:115 2022-03-29T17:50:01.376839+0000 0 ObjectCleanRegions clean_offsets: [0~197148,262684~18446744073709288931], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,422 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'76 lcod 11'76 mlcod 11'76 active add_log_entry 11'77 (11'76) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:115 2022-03-29T17:50:01.376839+0000 0 ObjectCleanRegions clean_offsets: [0~197148,262684~18446744073709288931], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,422 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'77 lcod 11'76 mlcod 11'76 active append_log approx pg log length = 77 2022-03-29T17:50:01.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,422 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'77 lcod 11'76 mlcod 11'76 active append_log transaction_applied = 1 2022-03-29T17:50:01.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,422 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,422 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'77, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,422 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,421 [shard 0] osd - client_request(id=117, detail=m=[osd_op(client.4125.0:116 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,423 [shard 0] osd - client_request(id=117, detail=m=[osd_op(client.4125.0:116 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,423 [shard 0] osd - OpSequencer::start_op: op=117, last_started=116, last_unblocked=116, last_completed=111 2022-03-29T17:50:01.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,424 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131600~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,424 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,424 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,424 [shard 0] osd - OpSequencer::finish_op_in_order: op=112, last_started=117, last_unblocked=117, last_completed=111 2022-03-29T17:50:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - client_request(id=118, detail=m=[osd_op(client.4125.0:117 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - client_request(id=118, detail=m=[osd_op(client.4125.0:117 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - client_request(id=117, detail=m=[osd_op(client.4125.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 262684~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - client_request(id=112, detail=m=[osd_op(client.4125.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131600~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'73 lua=0'0 crt=11'77 lcod 11'73 mlcod 11'73 active+clean calc_trim_to_aggressive limit = 11'73 2022-03-29T17:50:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'73 lua=0'0 crt=11'77 lcod 11'73 mlcod 11'73 active+clean append_log log((0'0,11'77], crt=11'77) {11'78 (11'77) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:116 2022-03-29T17:50:01.376855+0000 0 ObjectCleanRegions clean_offsets: [0~262684,262688~18446744073709288927], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'73 lua=0'0 crt=11'77 lcod 11'73 mlcod 11'73 active+clean add_log_entry 11'78 (11'77) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:116 2022-03-29T17:50:01.376855+0000 0 ObjectCleanRegions clean_offsets: [0~262684,262688~18446744073709288927], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'73 lua=0'0 crt=11'78 lcod 11'73 mlcod 11'73 active+clean append_log approx pg log length = 78 2022-03-29T17:50:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'73 lua=0'0 crt=11'78 lcod 11'73 mlcod 11'73 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,425 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'78, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,426 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,426 [shard 0] osd - client_request(id=118, detail=m=[osd_op(client.4125.0:117 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,426 [shard 0] osd - OpSequencer::start_op: op=118, last_started=117, last_unblocked=117, last_completed=112 2022-03-29T17:50:01.536 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,427 [shard 0] osd - replicated_request(id=78, detail=RepRequest(from=1 req=osd_repop(client.4125.0:116 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'78, mlcod=11'78) v3)) start 2022-03-29T17:50:01.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,427 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'77 (0'0,11'77] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'77 mlcod 11'77 active append_log log((0'0,11'77], crt=11'77) {11'78 (11'77) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:116 2022-03-29T17:50:01.376855+0000 0 ObjectCleanRegions clean_offsets: [0~262684,262688~18446744073709288927], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,427 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'77 lcod 11'77 mlcod 11'77 active add_log_entry 11'78 (11'77) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:116 2022-03-29T17:50:01.376855+0000 0 ObjectCleanRegions clean_offsets: [0~262684,262688~18446744073709288927], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,427 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'78 lcod 11'77 mlcod 11'77 active append_log approx pg log length = 78 2022-03-29T17:50:01.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,427 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'78 lcod 11'77 mlcod 11'77 active append_log transaction_applied = 1 2022-03-29T17:50:01.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,427 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,427 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'78, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,427 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,428 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,428 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131604~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - OpSequencer::finish_op_in_order: op=113, last_started=118, last_unblocked=118, last_completed=112 2022-03-29T17:50:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - client_request(id=118, detail=m=[osd_op(client.4125.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - client_request(id=113, detail=m=[osd_op(client.4125.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131604~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:50:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,429 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op setxattr 2022-03-29T17:50:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - handling op setxattr on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,429 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - setxattr on obj=1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'74 lua=0'0 crt=11'78 lcod 11'74 mlcod 11'74 active+clean calc_trim_to_aggressive limit = 11'74 2022-03-29T17:50:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'74 lua=0'0 crt=11'78 lcod 11'74 mlcod 11'74 active+clean append_log log((0'0,11'78], crt=11'78) {11'79 (11'78) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:117 2022-03-29T17:50:01.376876+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'74 lua=0'0 crt=11'78 lcod 11'74 mlcod 11'74 active+clean add_log_entry 11'79 (11'78) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:117 2022-03-29T17:50:01.376876+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'74 lua=0'0 crt=11'79 lcod 11'74 mlcod 11'74 active+clean append_log approx pg log length = 79 2022-03-29T17:50:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'74 lua=0'0 crt=11'79 lcod 11'74 mlcod 11'74 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'79, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,429 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.548 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,430 [shard 0] osd - replicated_request(id=79, detail=RepRequest(from=1 req=osd_repop(client.4125.0:117 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'79, mlcod=11'79) v3)) start 2022-03-29T17:50:01.548 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,430 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'78 (0'0,11'78] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'78 mlcod 11'78 active append_log log((0'0,11'78], crt=11'78) {11'79 (11'78) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:117 2022-03-29T17:50:01.376876+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.548 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,430 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'78 lcod 11'78 mlcod 11'78 active add_log_entry 11'79 (11'78) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:117 2022-03-29T17:50:01.376876+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.549 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,430 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'79 lcod 11'78 mlcod 11'78 active append_log approx pg log length = 79 2022-03-29T17:50:01.549 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,430 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'79 lcod 11'78 mlcod 11'78 active append_log transaction_applied = 1 2022-03-29T17:50:01.549 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,430 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.549 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,431 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'79, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,431 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,431 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197140~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,432 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,432 [shard 0] osd - OpSequencer::finish_op_in_order: op=114, last_started=118, last_unblocked=118, last_completed=113 2022-03-29T17:50:01.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,432 [shard 0] osd - client_request(id=114, detail=m=[osd_op(client.4125.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197140~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,433 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197144~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,433 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,433 [shard 0] osd - OpSequencer::finish_op_in_order: op=115, last_started=118, last_unblocked=118, last_completed=114 2022-03-29T17:50:01.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,434 [shard 0] osd - client_request(id=115, detail=m=[osd_op(client.4125.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197144~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,434 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197148~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,434 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,434 [shard 0] osd - OpSequencer::finish_op_in_order: op=116, last_started=118, last_unblocked=118, last_completed=115 2022-03-29T17:50:01.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,434 [shard 0] osd - client_request(id=116, detail=m=[osd_op(client.4125.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 197148~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,434 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 262684~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,435 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,435 [shard 0] osd - OpSequencer::finish_op_in_order: op=117, last_started=118, last_unblocked=118, last_completed=116 2022-03-29T17:50:01.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,435 [shard 0] osd - client_request(id=117, detail=m=[osd_op(client.4125.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 262684~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,435 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,435 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,435 [shard 0] osd - OpSequencer::finish_op_in_order: op=118, last_started=118, last_unblocked=118, last_completed=117 2022-03-29T17:50:01.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,435 [shard 0] osd - client_request(id=118, detail=m=[osd_op(client.4125.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {setxattr striper.size (16) in=28b, setxattr striper.version (16) in=31b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - client_request(id=119, detail=m=[osd_op(client.4125.0:118 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - client_request(id=119, detail=m=[osd_op(client.4125.0:118 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - client_request(id=119, detail=m=[osd_op(client.4125.0:118 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - OpSequencer::start_op: op=119, last_started=118, last_unblocked=118, last_completed=118 2022-03-29T17:50:01.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - client_request(id=120, detail=m=[osd_op(client.4125.0:119 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - client_request(id=120, detail=m=[osd_op(client.4125.0:119 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - client_request(id=119, detail=m=[osd_op(client.4125.0:118 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:01.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - client_request(id=120, detail=m=[osd_op(client.4125.0:119 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - OpSequencer::start_op: op=120, last_started=119, last_unblocked=119, last_completed=118 2022-03-29T17:50:01.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,477 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,478 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'79 mlcod 11'79 active+clean calc_trim_to_aggressive limit = 11'79 2022-03-29T17:50:01.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,478 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'79 mlcod 11'79 active+clean append_log log((0'0,11'79], crt=11'79) {11'80 (11'65) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:118 2022-03-29T17:50:01.476540+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,478 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'79 lua=0'0 crt=11'79 lcod 11'79 mlcod 11'79 active+clean add_log_entry 11'80 (11'65) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:118 2022-03-29T17:50:01.476540+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,478 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'79 lua=0'0 crt=11'80 lcod 11'79 mlcod 11'79 active+clean append_log approx pg log length = 80 2022-03-29T17:50:01.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,478 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'79 lua=0'0 crt=11'80 lcod 11'79 mlcod 11'79 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,478 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,478 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'80, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,478 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.565 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,479 [shard 0] osd - replicated_request(id=80, detail=RepRequest(from=1 req=osd_repop(client.4125.0:118 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'80, mlcod=11'80) v3)) start 2022-03-29T17:50:01.565 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,479 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'79 (0'0,11'79] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'79 mlcod 11'79 active append_log log((0'0,11'79], crt=11'79) {11'80 (11'65) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:118 2022-03-29T17:50:01.476540+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.565 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,479 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'79 lcod 11'79 mlcod 11'79 active add_log_entry 11'80 (11'65) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:118 2022-03-29T17:50:01.476540+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.566 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,479 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'80 lcod 11'79 mlcod 11'79 active append_log approx pg log length = 80 2022-03-29T17:50:01.566 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,479 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'80 lcod 11'79 mlcod 11'79 active append_log transaction_applied = 1 2022-03-29T17:50:01.566 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,479 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.566 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,479 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'80, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.567 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,479 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - client_request(id=120, detail=m=[osd_op(client.4125.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 131072~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:01.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'79 lua=0'0 crt=11'80 lcod 11'79 mlcod 11'79 active+clean calc_trim_to_aggressive limit = 11'79 2022-03-29T17:50:01.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,481 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'79 lua=0'0 crt=11'80 lcod 11'79 mlcod 11'79 active+clean append_log log((0'0,11'80], crt=11'80) {11'81 (11'80) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:119 2022-03-29T17:50:01.476636+0000 0 ObjectCleanRegions clean_offsets: [0~131072,196608~18446744073709355007], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,482 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'79 lua=0'0 crt=11'80 lcod 11'79 mlcod 11'79 active+clean add_log_entry 11'81 (11'80) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:119 2022-03-29T17:50:01.476636+0000 0 ObjectCleanRegions clean_offsets: [0~131072,196608~18446744073709355007], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,482 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'79 lua=0'0 crt=11'81 lcod 11'79 mlcod 11'79 active+clean append_log approx pg log length = 81 2022-03-29T17:50:01.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,482 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'79 lua=0'0 crt=11'81 lcod 11'79 mlcod 11'79 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,482 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,482 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'81, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,482 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,482 [shard 0] osd - client_request(id=121, detail=m=[osd_op(client.4125.0:120 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,482 [shard 0] osd - client_request(id=121, detail=m=[osd_op(client.4125.0:120 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,483 [shard 0] osd - replicated_request(id=81, detail=RepRequest(from=1 req=osd_repop(client.4125.0:119 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'81, mlcod=11'81) v3)) start 2022-03-29T17:50:01.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,483 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'80 (0'0,11'80] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'80 mlcod 11'80 active append_log log((0'0,11'80], crt=11'80) {11'81 (11'80) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:119 2022-03-29T17:50:01.476636+0000 0 ObjectCleanRegions clean_offsets: [0~131072,196608~18446744073709355007], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,483 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'80 lcod 11'80 mlcod 11'80 active add_log_entry 11'81 (11'80) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:119 2022-03-29T17:50:01.476636+0000 0 ObjectCleanRegions clean_offsets: [0~131072,196608~18446744073709355007], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,483 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'81 lcod 11'80 mlcod 11'80 active append_log approx pg log length = 81 2022-03-29T17:50:01.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,483 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'81 lcod 11'80 mlcod 11'80 active append_log transaction_applied = 1 2022-03-29T17:50:01.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,483 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,483 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'81, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.576 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,483 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,484 [shard 0] osd - client_request(id=121, detail=m=[osd_op(client.4125.0:120 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,484 [shard 0] osd - OpSequencer::start_op: op=121, last_started=120, last_unblocked=120, last_completed=118 2022-03-29T17:50:01.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,485 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:118 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - OpSequencer::finish_op_in_order: op=119, last_started=121, last_unblocked=121, last_completed=118 2022-03-29T17:50:01.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - client_request(id=122, detail=m=[osd_op(client.4125.0:121 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - client_request(id=122, detail=m=[osd_op(client.4125.0:121 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - client_request(id=119, detail=m=[osd_op(client.4125.0:118 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - client_request(id=121, detail=m=[osd_op(client.4125.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 196608~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - client_request(id=122, detail=m=[osd_op(client.4125.0:121 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - OpSequencer::start_op: op=122, last_started=121, last_unblocked=121, last_completed=119 2022-03-29T17:50:01.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:01.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'80 lua=0'0 crt=11'81 lcod 11'80 mlcod 11'80 active+clean calc_trim_to_aggressive limit = 11'80 2022-03-29T17:50:01.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'80 lua=0'0 crt=11'81 lcod 11'80 mlcod 11'80 active+clean append_log log((0'0,11'81], crt=11'81) {11'82 (11'81) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:120 2022-03-29T17:50:01.476676+0000 0 ObjectCleanRegions clean_offsets: [0~196608,262144~18446744073709289471], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'80 lua=0'0 crt=11'81 lcod 11'80 mlcod 11'80 active+clean add_log_entry 11'82 (11'81) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:120 2022-03-29T17:50:01.476676+0000 0 ObjectCleanRegions clean_offsets: [0~196608,262144~18446744073709289471], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,486 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'80 lua=0'0 crt=11'82 lcod 11'80 mlcod 11'80 active+clean append_log approx pg log length = 82 2022-03-29T17:50:01.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,487 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'80 lua=0'0 crt=11'82 lcod 11'80 mlcod 11'80 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,487 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,487 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'82, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,487 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.585 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,488 [shard 0] osd - replicated_request(id=82, detail=RepRequest(from=1 req=osd_repop(client.4125.0:120 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'82, mlcod=11'82) v3)) start 2022-03-29T17:50:01.586 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,488 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'81 (0'0,11'81] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'81 mlcod 11'81 active append_log log((0'0,11'81], crt=11'81) {11'82 (11'81) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:120 2022-03-29T17:50:01.476676+0000 0 ObjectCleanRegions clean_offsets: [0~196608,262144~18446744073709289471], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.586 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,488 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'81 lcod 11'81 mlcod 11'81 active add_log_entry 11'82 (11'81) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:120 2022-03-29T17:50:01.476676+0000 0 ObjectCleanRegions clean_offsets: [0~196608,262144~18446744073709289471], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.586 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,488 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'82 lcod 11'81 mlcod 11'81 active append_log approx pg log length = 82 2022-03-29T17:50:01.586 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,488 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'82 lcod 11'81 mlcod 11'81 active append_log transaction_applied = 1 2022-03-29T17:50:01.587 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,488 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.587 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,488 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'82, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.587 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,488 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,489 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,489 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 131072~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,489 [shard 0] osd - client_request(id=122, detail=m=[osd_op(client.4125.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 262144~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,489 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,490 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,490 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:01.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,490 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,490 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,490 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,490 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'81 lua=0'0 crt=11'82 lcod 11'81 mlcod 11'81 active+clean calc_trim_to_aggressive limit = 11'81 2022-03-29T17:50:01.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,490 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'81 lua=0'0 crt=11'82 lcod 11'81 mlcod 11'81 active+clean append_log log((0'0,11'82], crt=11'82) {11'83 (11'82) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:121 2022-03-29T17:50:01.476699+0000 0 ObjectCleanRegions clean_offsets: [0~262144,327680~18446744073709223935], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,490 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'81 lua=0'0 crt=11'82 lcod 11'81 mlcod 11'81 active+clean add_log_entry 11'83 (11'82) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:121 2022-03-29T17:50:01.476699+0000 0 ObjectCleanRegions clean_offsets: [0~262144,327680~18446744073709223935], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,491 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'81 lua=0'0 crt=11'83 lcod 11'81 mlcod 11'81 active+clean append_log approx pg log length = 83 2022-03-29T17:50:01.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,491 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'81 lua=0'0 crt=11'83 lcod 11'81 mlcod 11'81 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,491 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,491 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'83, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,491 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,491 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,491 [shard 0] osd - OpSequencer::finish_op_in_order: op=120, last_started=122, last_unblocked=122, last_completed=119 2022-03-29T17:50:01.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,491 [shard 0] osd - client_request(id=120, detail=m=[osd_op(client.4125.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 131072~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,492 [shard 0] osd - replicated_request(id=83, detail=RepRequest(from=1 req=osd_repop(client.4125.0:121 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'83, mlcod=11'83) v3)) start 2022-03-29T17:50:01.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,492 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'82 (0'0,11'82] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'82 mlcod 11'82 active append_log log((0'0,11'82], crt=11'82) {11'83 (11'82) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:121 2022-03-29T17:50:01.476699+0000 0 ObjectCleanRegions clean_offsets: [0~262144,327680~18446744073709223935], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,492 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'82 lcod 11'82 mlcod 11'82 active add_log_entry 11'83 (11'82) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:121 2022-03-29T17:50:01.476699+0000 0 ObjectCleanRegions clean_offsets: [0~262144,327680~18446744073709223935], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,492 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'83 lcod 11'82 mlcod 11'82 active append_log approx pg log length = 83 2022-03-29T17:50:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,492 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'83 lcod 11'82 mlcod 11'82 active append_log transaction_applied = 1 2022-03-29T17:50:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,492 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,492 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'83, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,493 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,494 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 196608~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,494 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,494 [shard 0] osd - OpSequencer::finish_op_in_order: op=121, last_started=122, last_unblocked=122, last_completed=120 2022-03-29T17:50:01.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,494 [shard 0] osd - client_request(id=121, detail=m=[osd_op(client.4125.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 196608~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,495 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 262144~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,495 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,495 [shard 0] osd - OpSequencer::finish_op_in_order: op=122, last_started=122, last_unblocked=122, last_completed=121 2022-03-29T17:50:01.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,495 [shard 0] osd - client_request(id=122, detail=m=[osd_op(client.4125.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 262144~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,526 [shard 0] osd - client_request(id=123, detail=m=[osd_op(client.4125.0:122 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,526 [shard 0] osd - client_request(id=123, detail=m=[osd_op(client.4125.0:122 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - client_request(id=123, detail=m=[osd_op(client.4125.0:122 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - OpSequencer::start_op: op=123, last_started=122, last_unblocked=122, last_completed=122 2022-03-29T17:50:01.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - client_request(id=123, detail=m=[osd_op(client.4125.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:01.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:01.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:01.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:01.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'83 mlcod 11'83 active+clean calc_trim_to_aggressive limit = 11'83 2022-03-29T17:50:01.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'83 mlcod 11'83 active+clean append_log log((0'0,11'83], crt=11'83) {11'84 (11'79) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:122 2022-03-29T17:50:01.526526+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'83 lua=0'0 crt=11'83 lcod 11'83 mlcod 11'83 active+clean add_log_entry 11'84 (11'79) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:122 2022-03-29T17:50:01.526526+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'83 lua=0'0 crt=11'84 lcod 11'83 mlcod 11'83 active+clean append_log approx pg log length = 84 2022-03-29T17:50:01.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'83 lua=0'0 crt=11'84 lcod 11'83 mlcod 11'83 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'84, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,527 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,535 [shard 0] osd - replicated_request(id=84, detail=RepRequest(from=1 req=osd_repop(client.4125.0:122 1.0 e11/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 11'84, mlcod=11'84) v3)) start 2022-03-29T17:50:01.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,535 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'83 (0'0,11'83] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'83 mlcod 11'83 active append_log log((0'0,11'83], crt=11'83) {11'84 (11'79) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:122 2022-03-29T17:50:01.526526+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,535 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'83 lcod 11'83 mlcod 11'83 active add_log_entry 11'84 (11'79) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:122 2022-03-29T17:50:01.526526+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,535 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'84 lcod 11'83 mlcod 11'83 active append_log approx pg log length = 84 2022-03-29T17:50:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,536 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'84 lcod 11'83 mlcod 11'83 active append_log transaction_applied = 1 2022-03-29T17:50:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,536 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,536 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'84, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,536 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,576 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:01.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,576 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:01.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - OpSequencer::finish_op_in_order: op=123, last_started=123, last_unblocked=123, last_completed=122 2022-03-29T17:50:01.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - client_request(id=123, detail=m=[osd_op(client.4125.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - client_request(id=124, detail=m=[osd_op(client.4125.0:123 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - client_request(id=124, detail=m=[osd_op(client.4125.0:123 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - client_request(id=124, detail=m=[osd_op(client.4125.0:123 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - OpSequencer::start_op: op=124, last_started=123, last_unblocked=123, last_completed=123 2022-03-29T17:50:01.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - client_request(id=124, detail=m=[osd_op(client.4125.0:123 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,577 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:01.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:50:01.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:01.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.617 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,578 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:01.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:50:01.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.618 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,578 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,578 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:50:01.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'84 mlcod 11'84 active+clean calc_trim_to_aggressive limit = 11'84 2022-03-29T17:50:01.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'84 mlcod 11'84 active+clean append_log log((0'0,11'84], crt=11'84) {11'85 (11'83) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:123 2022-03-29T17:50:01.576903+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'84 lua=0'0 crt=11'84 lcod 11'84 mlcod 11'84 active+clean add_log_entry 11'85 (11'83) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:123 2022-03-29T17:50:01.576903+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'84 lua=0'0 crt=11'85 lcod 11'84 mlcod 11'84 active+clean append_log approx pg log length = 85 2022-03-29T17:50:01.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'84 lua=0'0 crt=11'85 lcod 11'84 mlcod 11'84 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'85, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,579 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,617 [shard 0] osd - replicated_request(id=85, detail=RepRequest(from=1 req=osd_repop(client.4125.0:123 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'85, mlcod=11'85) v3)) start 2022-03-29T17:50:01.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,617 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'84 (0'0,11'84] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'84 mlcod 11'84 active append_log log((0'0,11'84], crt=11'84) {11'85 (11'83) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:123 2022-03-29T17:50:01.576903+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,617 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'84 lcod 11'84 mlcod 11'84 active add_log_entry 11'85 (11'83) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:123 2022-03-29T17:50:01.576903+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,617 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'85 lcod 11'84 mlcod 11'84 active append_log approx pg log length = 85 2022-03-29T17:50:01.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,617 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'85 lcod 11'84 mlcod 11'84 active append_log transaction_applied = 1 2022-03-29T17:50:01.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,617 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,618 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'85, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,618 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,658 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:123 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:01.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,659 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,659 [shard 0] osd - OpSequencer::finish_op_in_order: op=124, last_started=124, last_unblocked=124, last_completed=123 2022-03-29T17:50:01.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,659 [shard 0] osd - client_request(id=124, detail=m=[osd_op(client.4125.0:123 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:01.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,662 [shard 0] osd - client_request(id=125, detail=m=[osd_op(client.4125.0:124 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:01.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,662 [shard 0] osd - client_request(id=125, detail=m=[osd_op(client.4125.0:124 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:01.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,662 [shard 0] osd - client_request(id=125, detail=m=[osd_op(client.4125.0:124 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:01.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - OpSequencer::start_op: op=125, last_started=124, last_unblocked=124, last_completed=124 2022-03-29T17:50:01.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - client_request(id=125, detail=m=[osd_op(client.4125.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:01.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:01.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:01.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:50:01.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,663 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.677 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,664 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:01.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:50:01.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:01.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - cmpxattr lhs=, rhs= 2022-03-29T17:50:01.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:01.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:01.681 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,664 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:01.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:01.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:01.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:01.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'85 mlcod 11'85 active+clean calc_trim_to_aggressive limit = 11'85 2022-03-29T17:50:01.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'85 mlcod 11'85 active+clean append_log log((0'0,11'85], crt=11'85) {11'86 (11'85) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:124 2022-03-29T17:50:01.662538+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'85 lcod 11'85 mlcod 11'85 active+clean add_log_entry 11'86 (11'85) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:124 2022-03-29T17:50:01.662538+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean append_log approx pg log length = 86 2022-03-29T17:50:01.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean append_log transaction_applied = 1 2022-03-29T17:50:01.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'86, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,664 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:01.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,699 [shard 0] osd - replicated_request(id=86, detail=RepRequest(from=1 req=osd_repop(client.4125.0:124 1.0 e11/8 1:30306672:devicehealth::main.db.0000000000000000:head v 11'86, mlcod=11'86) v3)) start 2022-03-29T17:50:01.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,699 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'85 (0'0,11'85] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'85 mlcod 11'85 active append_log log((0'0,11'85], crt=11'85) {11'86 (11'85) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:124 2022-03-29T17:50:01.662538+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:01.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,699 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'85 lcod 11'85 mlcod 11'85 active add_log_entry 11'86 (11'85) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:124 2022-03-29T17:50:01.662538+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:01.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,699 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active append_log approx pg log length = 86 2022-03-29T17:50:01.703 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,700 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active append_log transaction_applied = 1 2022-03-29T17:50:01.703 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,700 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:01.703 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,700 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 11'86, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.703 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,700 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:01.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,711 [shard 0] osd - handle_osd_map osd_map(11..12 src has 1..12) v4 2022-03-29T17:50:01.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,711 [shard 0] osd - handle_osd_map epochs [11..12], i have 11, src has [1..12] 2022-03-29T17:50:01.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,712 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:01.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,713 [shard 0] osd - handle_osd_map osd_map(12..12 src has 1..12) v4 2022-03-29T17:50:01.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,713 [shard 0] osd - handle_osd_map epochs [12..12], i have 11, src has [1..12] 2022-03-29T17:50:01.715 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,714 [shard 0] osd - handle_osd_map osd_map(12..12 src has 1..12) v4 2022-03-29T17:50:01.716 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,714 [shard 0] osd - handle_osd_map epochs [12..12], i have 11, src has [1..12] 2022-03-29T17:50:01.717 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,715 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:01.717 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,716 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:01.719 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,716 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e12 {2.2=11,2022-03-29T17:50:00.701078+0000}) v2): starting 2022-03-29T17:50:01.719 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,716 [shard 0] osd - handle_pg_create: 2.2 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:01.719 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,716 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=? pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): start 2022-03-29T17:50:01.720 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,717 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e12 {2.2=11,2022-03-29T17:50:00.701078+0000}) v2): about to fork future 2022-03-29T17:50:01.720 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,717 [shard 0] osd - osd.0: committed_osd_maps(12, 12) 2022-03-29T17:50:01.720 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,717 [shard 0] osd - osd.0: map e 12 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:01.721 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,717 [shard 0] osd - evt epoch is 12, i have 11, will wait 2022-03-29T17:50:01.721 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,717 [shard 0] osd - osd.2: committed_osd_maps(12, 12) 2022-03-29T17:50:01.722 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,717 [shard 0] osd - osd.2: map e 12 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:01.722 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rbd' on pool 'rbd' 2022-03-29T17:50:01.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,717 [shard 0] osd - pg_advance_map(id=3, detail=PGAdvanceMap(pg=2.0 from=11 to=12)): start 2022-03-29T17:50:01.724 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,717 [shard 0] osd - osdmap_subscribe(11) 2022-03-29T17:50:01.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,717 [shard 0] osd - pg_advance_map(id=4, detail=PGAdvanceMap(pg=2.1 from=11 to=12)): start 2022-03-29T17:50:01.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,718 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:50:01.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,718 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:50:01.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,718 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 11, need_up_thru now false 2022-03-29T17:50:01.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,718 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:50:01.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,718 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:50:01.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,718 [shard 0] osd - pg_advance_map(id=4, detail=PGAdvanceMap(pg=1.0 from=11 to=12)): start 2022-03-29T17:50:01.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,718 [shard 0] osd - pg_advance_map(id=5, detail=PGAdvanceMap(pg=2.2 from=11 to=12)): start 2022-03-29T17:50:01.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:50:01.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:50:01.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 11, need_up_thru now false 2022-03-29T17:50:01.731 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:50:01.731 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:50:01.731 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 11, need_up_thru now false 2022-03-29T17:50:01.732 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:50:01.732 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:50:01.732 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:01.732 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:01.733 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.733 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:2 2022-03-29T17:50:01.733 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.935757 2 0.000934 2022-03-29T17:50:01.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:50:01.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:50:01.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,719 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 mlcod 11'86 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:01.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 mlcod 11'86 active state: Started advmap 2022-03-29T17:50:01.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 mlcod 11'86 active check_recovery_sources no source osds () went down 2022-03-29T17:50:01.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:01.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:01.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.740 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,720 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576200.783958, 0.000934443 spent on 2 events 2022-03-29T17:50:01.741 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:50:01.741 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.003469 0 0.000000 2022-03-29T17:50:01.742 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,720 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576200.7169044, 0.0 spent on 0 events 2022-03-29T17:50:01.742 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:50:01.743 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,720 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:50:01.743 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:50:01.743 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:50:01.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:2 2022-03-29T17:50:01.746 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.944684 2 0.001431 2022-03-29T17:50:01.747 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,720 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576200.7761457, 0.001431796 spent on 2 events 2022-03-29T17:50:01.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:50:01.747 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,720 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.003668 0 0.000000 2022-03-29T17:50:01.748 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,721 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576200.717278, 0.0 spent on 0 events 2022-03-29T17:50:01.748 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:50:01.748 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,721 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:50:01.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:50:01.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:50:01.750 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 24.353706360s) 2022-03-29T17:50:01.750 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:50:01.751 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.751 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:50:01.751 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.751 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:50:01.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 24.330034256s) 2022-03-29T17:50:01.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:50:01.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:50:01.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,721 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:50:01.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:50:01.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:50:01.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:50:01.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:50:01.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:50:01.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:50:01.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:01.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:01.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:01.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:01.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:01.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:01.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:01.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:01.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:50:01.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:01.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:01.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:01.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:01.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:01.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:01.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:3 2022-03-29T17:50:01.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:50:01.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,722 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:01.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:3 2022-03-29T17:50:01.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:50:01.767 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:50:01.767 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,723 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:50:01.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.770 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:50:01.770 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,723 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:50:01.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,723 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 mlcod 11'86 active activate_map 2022-03-29T17:50:01.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:01.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:01.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:2 2022-03-29T17:50:01.774 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.938814 2 0.000755 2022-03-29T17:50:01.774 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,724 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576200.7856874, 0.000755653 spent on 2 events 2022-03-29T17:50:01.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:50:01.775 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.002633 0 0.000000 2022-03-29T17:50:01.775 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,724 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576200.7219908, 0.0 spent on 0 events 2022-03-29T17:50:01.775 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:50:01.775 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,724 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:50:01.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:50:01.776 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 mlcod 11'86 active activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.777 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.777 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,724 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.777 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,725 [shard 0] osd - handle_osd_map osd_map(11..12 src has 1..12) v4 2022-03-29T17:50:01.777 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,725 [shard 0] osd - handle_osd_map epochs [11..12], i have 12, src has [1..12] 2022-03-29T17:50:01.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:50:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 24.358369827s) 2022-03-29T17:50:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:50:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:50:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.1 2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:50:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:50:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:50:01.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,725 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:50:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:01.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:01.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:01.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:3 2022-03-29T17:50:01.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:50:01.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_advance_map(id=4, detail=PGAdvanceMap(pg=1.0 from=11 to=12)): complete 2022-03-29T17:50:01.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.2 2022-03-29T17:50:01.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,726 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 12 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 11 2022-03-29T17:50:01.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:01.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - pg_advance_map(id=5, detail=PGAdvanceMap(pg=2.2 from=11 to=12)): complete 2022-03-29T17:50:01.790 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,727 [shard 0] osd - osd.0: now active 2022-03-29T17:50:01.791 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,727 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:50:01.791 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,727 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:50:01.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=? pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): got map 12 2022-03-29T17:50:01.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,727 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=? pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): pg present 2022-03-29T17:50:01.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,728 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:01.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,728 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 NullEvt +create_info for pg: 2.2 2022-03-29T17:50:01.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,728 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,728 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,728 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,728 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=? pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:01.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,728 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.796 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,728 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e12 {2.2=11,2022-03-29T17:50:00.701078+0000}) v2): sub events complete 2022-03-29T17:50:01.796 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,729 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e12 {2.2=11,2022-03-29T17:50:00.701078+0000}) v2): complete 2022-03-29T17:50:01.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,729 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,729 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,729 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.0 2022-03-29T17:50:01.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,729 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 12 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 11 2022-03-29T17:50:01.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,729 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,729 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:01.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,729 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,729 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=? pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): complete 2022-03-29T17:50:01.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,730 [shard 0] osd - pg_advance_map(id=4, detail=PGAdvanceMap(pg=2.1 from=11 to=12)): complete 2022-03-29T17:50:01.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,730 [shard 0] osd - pg_advance_map(id=3, detail=PGAdvanceMap(pg=2.0 from=11 to=12)): complete 2022-03-29T17:50:01.801 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,730 [shard 0] osd - osd.2: now active 2022-03-29T17:50:01.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,730 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.1 2022-03-29T17:50:01.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,730 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 12 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 11 2022-03-29T17:50:01.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,730 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,730 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:01.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,730 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,730 [shard 0] osd - peering_event(id=8, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:01.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,731 [shard 0] osd - peering_event(id=9, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:01.804 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,731 [shard 0] osd - osd.1: committed_osd_maps(12, 12) 2022-03-29T17:50:01.805 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,731 [shard 0] osd - osd.1: map e 12 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:01.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:01.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.353706360s int 16.000000000s) +create_info)): start 2022-03-29T17:50:01.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - pg_advance_map(id=10, detail=PGAdvanceMap(pg=1.0 from=11 to=12)): start 2022-03-29T17:50:01.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - pg_advance_map(id=11, detail=PGAdvanceMap(pg=2.3 from=11 to=12)): start 2022-03-29T17:50:01.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - pg_advance_map(id=12, detail=PGAdvanceMap(pg=2.4 from=11 to=12)): start 2022-03-29T17:50:01.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - pg_advance_map(id=13, detail=PGAdvanceMap(pg=2.5 from=11 to=12)): start 2022-03-29T17:50:01.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - pg_advance_map(id=14, detail=PGAdvanceMap(pg=2.6 from=11 to=12)): start 2022-03-29T17:50:01.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - pg_advance_map(id=15, detail=PGAdvanceMap(pg=2.7 from=11 to=12)): start 2022-03-29T17:50:01.808 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-03-29T17:50:01.809 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-29T17:50:01.809 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-03-29T17:50:01.809 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-03-29T17:50:01.810 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-03-29T17:50:01.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,732 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:01.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:50:01.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 11, need_up_thru now false 2022-03-29T17:50:01.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:50:01.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:50:01.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:01.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:01.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:2 2022-03-29T17:50:01.824 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.911274 2 0.000760 2022-03-29T17:50:01.825 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,733 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576200.8222387, 0.000760719 spent on 2 events 2022-03-29T17:50:01.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:50:01.825 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.986196 0 0.000000 2022-03-29T17:50:01.825 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,733 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576200.7473848, 0.0 spent on 0 events 2022-03-29T17:50:01.826 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:50:01.826 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,733 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:50:01.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:50:01.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:50:01.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 24.429607391s) 2022-03-29T17:50:01.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:50:01.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:50:01.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:50:01.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,733 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:50:01.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:50:01.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:50:01.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:01.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:01.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:01.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:01.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:01.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:01.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:01.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:01.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:3 2022-03-29T17:50:01.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:50:01.832 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:50:01.832 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,734 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:50:01.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 11 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:01.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean state: Active advmap 2022-03-29T17:50:01.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean state: Started advmap 2022-03-29T17:50:01.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:01.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:01.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:50:01.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 11, need_up_thru now false 2022-03-29T17:50:01.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:50:01.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,734 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:50:01.835 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,734 [shard 0] osd - handle_osd_map osd_map(12..12 src has 1..12) v4 2022-03-29T17:50:01.835 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,734 [shard 0] osd - handle_osd_map epochs [12..12], i have 12, src has [1..12] 2022-03-29T17:50:01.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:01.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:50:01.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 11, need_up_thru now false 2022-03-29T17:50:01.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:50:01.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:50:01.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean activate_map 2022-03-29T17:50:01.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean state: Active: handling ActMap 2022-03-29T17:50:01.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean state: handle ActMap primary 2022-03-29T17:50:01.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean publish_stats_to_osd 11:11 2022-03-29T17:50:01.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'85 lua=0'0 crt=11'86 lcod 11'85 mlcod 11'85 active+clean activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:01.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:01.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:2 2022-03-29T17:50:01.839 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.918194 2 0.001139 2022-03-29T17:50:01.840 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,735 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576200.8172772, 0.001139053 spent on 2 events 2022-03-29T17:50:01.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:50:01.840 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.997723 0 0.000000 2022-03-29T17:50:01.840 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,735 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576200.7378008, 0.0 spent on 0 events 2022-03-29T17:50:01.840 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:50:01.841 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,735 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:50:01.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:50:01.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:50:01.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 24.431535721s) 2022-03-29T17:50:01.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:50:01.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:50:01.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:50:01.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:50:01.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:50:01.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:50:01.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:01.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:01.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,735 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:01.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:3 2022-03-29T17:50:01.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:50:01.846 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,736 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:50:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,736 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:50:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:01.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.429607391s int 16.000000000s) +create_info)): start 2022-03-29T17:50:01.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.429607391s int 16.000000000s) +create_info)): got map 12 2022-03-29T17:50:01.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,736 [shard 0] osd - Creating 2.4 2022-03-29T17:50:01.858 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:01.858 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:01.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:01.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:01.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:50:01.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:01.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:50:01.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 8.346261978s + 0.000000000s) 2022-03-29T17:50:01.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_advance_map(id=6, detail=PGAdvanceMap(pg=2.4 from=11 to=12 do_init)): start 2022-03-29T17:50:01.860 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000453 0 0.000000 2022-03-29T17:50:01.860 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - Exiting state: Initial, entered at 1648576201.7371094, 0.0 spent on 0 events 2022-03-29T17:50:01.861 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:01.861 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:01.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:01.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:01.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:01.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:01.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:01.862 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000114 1 2022-03-29T17:50:01.737699+0000 2022-03-29T17:50:01.863 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - Exiting state: Reset, entered at 1648576201.7376025, 1648576201.737699 spent on 1 events 2022-03-29T17:50:01.863 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:01.863 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - Entering state: Started 2022-03-29T17:50:01.863 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:01.863 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - Entering state: Start 2022-03-29T17:50:01.864 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:50:01.864 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:50:01.864 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - Exiting state: Start, entered at 1648576201.7377768, 0.0 spent on 0 events 2022-03-29T17:50:01.864 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:50:01.865 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,737 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:50:01.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:01.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,737 [shard 0] osd - pg_epoch 11 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:01.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:50:01.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:50:01.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:01.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - Created 2.4 2022-03-29T17:50:01.867 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,738 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:50:01.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.429607391s int 16.000000000s) +create_info)): pg present 2022-03-29T17:50:01.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.429607391s int 16.000000000s) +create_info for pg: 2.4 2022-03-29T17:50:01.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:50:01.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 24.429607391s int 16.000000000s) 2022-03-29T17:50:01.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 24.347360611s 2022-03-29T17:50:01.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 24.347360611s 2022-03-29T17:50:01.869 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.000674 3 0.000215 2022-03-29T17:50:01.870 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,738 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576201.7378662, 0.000215468 spent on 3 events 2022-03-29T17:50:01.870 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:50:01.870 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,738 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:50:01.870 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:01.871 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,738 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:01.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:50:01.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:50:01.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.429607391s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:50:01.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:01.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,738 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:01.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.4 2022-03-29T17:50:01.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:50:01.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:01.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - pg_advance_map(id=6, detail=PGAdvanceMap(pg=2.4 from=11 to=12 do_init)): complete 2022-03-29T17:50:01.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.429607391s int 16.000000000s) +create_info)): complete 2022-03-29T17:50:01.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.4 2022-03-29T17:50:01.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: react 12 telling primary 2022-03-29T17:50:01.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:01.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,739 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:01.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:01.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:01.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:2 2022-03-29T17:50:01.879 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.919903 2 0.000672 2022-03-29T17:50:01.879 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,740 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576200.8202543, 0.000672006 spent on 2 events 2022-03-29T17:50:01.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:50:01.880 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.995629 0 0.000000 2022-03-29T17:50:01.880 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,740 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576200.744583, 0.0 spent on 0 events 2022-03-29T17:50:01.880 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:50:01.880 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,740 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:50:01.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:50:01.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=0 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:01.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:50:01.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 24.436550140s) 2022-03-29T17:50:01.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:50:01.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:50:01.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:50:01.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:50:01.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:50:01.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:50:01.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:01.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:01.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:01.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:01.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,740 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:01.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:01.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:01.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:01.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:3 2022-03-29T17:50:01.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:50:01.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:50:01.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,741 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:50:01.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:01.891 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:01.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:50:01.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 11, need_up_thru now false 2022-03-29T17:50:01.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:50:01.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:50:01.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:50:01.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:50:01.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 11, need_up_thru now false 2022-03-29T17:50:01.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:50:01.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,741 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:50:01.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:01.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:01.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:2 2022-03-29T17:50:01.904 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.916022 2 0.000666 2022-03-29T17:50:01.905 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,742 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576200.82608, 0.000666369 spent on 2 events 2022-03-29T17:50:01.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:50:01.905 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.991359 0 0.000000 2022-03-29T17:50:01.905 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,742 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576200.7507987, 0.0 spent on 0 events 2022-03-29T17:50:01.906 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:50:01.906 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,742 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:50:01.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:50:01.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:50:01.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 24.438203812s) 2022-03-29T17:50:01.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:50:01.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.2 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:50:01.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.2 2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:50:01.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:50:01.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:50:01.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:50:01.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:01.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:01.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:50:01.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:01.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:01.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:01.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:01.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:01.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:3 2022-03-29T17:50:01.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:50:01.912 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:50:01.912 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,742 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:50:01.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,742 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:50:01.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:50:01.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 11:2 2022-03-29T17:50:01.914 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.924556 2 0.000984 2022-03-29T17:50:01.914 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,743 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576200.818741, 0.000984184 spent on 2 events 2022-03-29T17:50:01.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:50:01.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.002806 0 0.000000 2022-03-29T17:50:01.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,743 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576200.7405577, 0.0 spent on 0 events 2022-03-29T17:50:01.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:50:01.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,743 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:50:01.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:50:01.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:50:01.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 24.439384460s) 2022-03-29T17:50:01.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:50:01.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:50:01.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:50:01.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate peer osd.0 2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:50:01.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:50:01.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:50:01.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:50:01.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:01.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:01.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:01.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:01.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:01.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:01.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:01.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:01.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:01.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:3 2022-03-29T17:50:01.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:50:01.921 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:50:01.922 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,743 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:50:01.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,743 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,744 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,744 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,744 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,744 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:01.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,744 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.438203812s int 16.000000000s) +create_info)): start 2022-03-29T17:50:01.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,744 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.438203812s int 16.000000000s) +create_info)): got map 12 2022-03-29T17:50:01.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,744 [shard 0] osd - Creating 2.3 2022-03-29T17:50:01.932 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:01.932 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:01.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 2} acting {1, 2} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:01.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:01.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:50:01.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:01.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:50:01.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 8.378513336s + 0.000000000s) 2022-03-29T17:50:01.934 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.934 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_advance_map(id=5, detail=PGAdvanceMap(pg=2.3 from=11 to=12 do_init)): start 2022-03-29T17:50:01.934 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000414 0 0.000000 2022-03-29T17:50:01.934 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - Exiting state: Initial, entered at 1648576201.745217, 0.0 spent on 0 events 2022-03-29T17:50:01.935 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:01.935 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:01.935 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:01.935 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:01.935 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:01.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:01.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:01.936 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000113 1 2022-03-29T17:50:01.745769+0000 2022-03-29T17:50:01.936 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - Exiting state: Reset, entered at 1648576201.7456758, 1648576201.7457695 spent on 1 events 2022-03-29T17:50:01.937 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:01.937 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - Entering state: Started 2022-03-29T17:50:01.937 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:01.937 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - Entering state: Start 2022-03-29T17:50:01.938 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:50:01.938 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:50:01.938 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - Exiting state: Start, entered at 1648576201.7458704, 0.0 spent on 0 events 2022-03-29T17:50:01.938 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:50:01.939 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,745 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:50:01.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:01.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,745 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 11 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:50:01.940 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:50:01.940 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:50:01.940 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:01.940 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.940 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.941 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - Created 2.3 2022-03-29T17:50:01.941 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,746 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:50:01.941 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.941 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.438203812s int 16.000000000s) +create_info)): pg present 2022-03-29T17:50:01.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.438203812s int 16.000000000s) +create_info for pg: 2.3 2022-03-29T17:50:01.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:50:01.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 24.438203812s int 16.000000000s) 2022-03-29T17:50:01.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 24.379606247s 2022-03-29T17:50:01.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 24.379606247s 2022-03-29T17:50:01.943 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.000663 3 0.000240 2022-03-29T17:50:01.944 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,746 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576201.7459497, 0.000240387 spent on 3 events 2022-03-29T17:50:01.944 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:50:01.944 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,746 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:50:01.944 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:01.945 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,746 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:01.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:50:01.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:50:01.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.438203812s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:50:01.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:01.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,746 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:01.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.3 2022-03-29T17:50:01.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:50:01.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:01.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - pg_advance_map(id=12, detail=PGAdvanceMap(pg=2.4 from=11 to=12)): complete 2022-03-29T17:50:01.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - pg_advance_map(id=5, detail=PGAdvanceMap(pg=2.3 from=11 to=12 do_init)): complete 2022-03-29T17:50:01.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:01.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,747 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - peering_event(id=10, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.438203812s int 16.000000000s) +create_info)): complete 2022-03-29T17:50:01.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - peering_event(id=11, detail=PeeringEvent(from=2 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:01.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.3 2022-03-29T17:50:01.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: react 12 telling primary 2022-03-29T17:50:01.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:01.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:01.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - pg_advance_map(id=13, detail=PGAdvanceMap(pg=2.5 from=11 to=12)): complete 2022-03-29T17:50:01.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,748 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:01.955 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,749 [shard 0] osd - evt epoch is 12, i have 11, will wait 2022-03-29T17:50:01.955 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,749 [shard 0] osd - osdmap_subscribe(11) 2022-03-29T17:50:01.958 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,749 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:01.958 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,749 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.431535721s int 16.000000000s) +create_info)): start 2022-03-29T17:50:01.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,749 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.431535721s int 16.000000000s) +create_info)): got map 12 2022-03-29T17:50:01.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,749 [shard 0] osd - Creating 2.5 2022-03-29T17:50:01.959 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:01.959 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:01.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:01.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:01.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:50:01.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:01.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:50:01.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 8.359327316s + 0.000000000s) 2022-03-29T17:50:01.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_advance_map(id=7, detail=PGAdvanceMap(pg=2.5 from=11 to=12 do_init)): start 2022-03-29T17:50:01.962 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000431 0 0.000000 2022-03-29T17:50:01.962 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - Exiting state: Initial, entered at 1648576201.750183, 0.0 spent on 0 events 2022-03-29T17:50:01.962 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:01.962 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:01.964 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000116 1 2022-03-29T17:50:01.750754+0000 2022-03-29T17:50:01.964 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - Exiting state: Reset, entered at 1648576201.7506561, 1648576201.750754 spent on 1 events 2022-03-29T17:50:01.964 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,750 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:01.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,750 [shard 0] osd - peering_event(id=12, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:01.984 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - Entering state: Started 2022-03-29T17:50:01.985 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:01.985 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - Entering state: Start 2022-03-29T17:50:01.985 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:50:01.985 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000069 0 0.000000 2022-03-29T17:50:01.986 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - Exiting state: Start, entered at 1648576201.7512033, 0.0 spent on 0 events 2022-03-29T17:50:01.986 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:50:01.986 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:50:01.986 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:01.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 11 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:01.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:50:01.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:50:01.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:01.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:01.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - Created 2.5 2022-03-29T17:50:01.988 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,751 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:50:01.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:01.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.436550140s int 16.000000000s) +create_info)): start 2022-03-29T17:50:01.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,751 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.431535721s int 16.000000000s) +create_info)): pg present 2022-03-29T17:50:01.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.436550140s int 16.000000000s) +create_info)): got map 12 2022-03-29T17:50:01.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:01.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - Creating 2.7 2022-03-29T17:50:01.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:01.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.431535721s int 16.000000000s) +create_info for pg: 2.5 2022-03-29T17:50:01.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:50:01.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:50:01.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 24.431535721s int 16.000000000s) 2022-03-29T17:50:01.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 24.361429214s 2022-03-29T17:50:01.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 24.361429214s 2022-03-29T17:50:01.992 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.001301 3 0.000614 2022-03-29T17:50:01.993 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,752 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576201.7513063, 0.000614014 spent on 3 events 2022-03-29T17:50:01.993 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:50:01.993 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,752 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:50:01.993 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:01.994 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,752 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:01.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:50:01.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:01.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:50:01.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - log is not dirty 2022-03-29T17:50:01.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.431535721s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:50:01.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,752 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:01.995 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:01.995 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:01.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:01.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:01.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:50:01.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:01.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:50:01.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 8.362089157s + 0.000000000s) 2022-03-29T17:50:01.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:01.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_advance_map(id=8, detail=PGAdvanceMap(pg=2.7 from=11 to=12 do_init)): start 2022-03-29T17:50:01.998 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000465 0 0.000000 2022-03-29T17:50:01.998 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - Exiting state: Initial, entered at 1648576201.753001, 0.0 spent on 0 events 2022-03-29T17:50:01.998 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:01.998 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:01.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:01.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:01.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:01.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:02.000 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.000 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000112 1 2022-03-29T17:50:01.753596+0000 2022-03-29T17:50:02.000 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - Exiting state: Reset, entered at 1648576201.753507, 1648576201.7535963 spent on 1 events 2022-03-29T17:50:02.000 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:02.000 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - Entering state: Started 2022-03-29T17:50:02.001 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:02.001 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - Entering state: Start 2022-03-29T17:50:02.001 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:50:02.001 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000037 0 0.000000 2022-03-29T17:50:02.002 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - Exiting state: Start, entered at 1648576201.7536778, 0.0 spent on 0 events 2022-03-29T17:50:02.002 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:50:02.002 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,753 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:50:02.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 11 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:50:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:50:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,753 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:02.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:02.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - Created 2.7 2022-03-29T17:50:02.005 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,754 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:50:02.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.5 2022-03-29T17:50:02.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:50:02.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:02.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:02.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.436550140s int 16.000000000s) +create_info)): pg present 2022-03-29T17:50:02.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:02.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.439384460s int 16.000000000s) +create_info)): start 2022-03-29T17:50:02.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - pg_advance_map(id=7, detail=PGAdvanceMap(pg=2.5 from=11 to=12 do_init)): complete 2022-03-29T17:50:02.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,754 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:02.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:02.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.431535721s int 16.000000000s) +create_info)): complete 2022-03-29T17:50:02.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.5 2022-03-29T17:50:02.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: react 12 telling primary 2022-03-29T17:50:02.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:02.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.439384460s int 16.000000000s) +create_info)): got map 12 2022-03-29T17:50:02.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - pg_advance_map(id=8, detail=PGAdvanceMap(pg=2.7 from=11 to=12 do_init)): complete 2022-03-29T17:50:02.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=0 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:02.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - Creating 2.6 2022-03-29T17:50:02.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.436550140s int 16.000000000s) +create_info for pg: 2.7 2022-03-29T17:50:02.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:50:02.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,755 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 24.436550140s int 16.000000000s) 2022-03-29T17:50:02.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 24.364866257s 2022-03-29T17:50:02.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 24.364866257s 2022-03-29T17:50:02.012 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.002297 3 0.000197 2022-03-29T17:50:02.013 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,756 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576201.7537477, 0.00019791 spent on 3 events 2022-03-29T17:50:02.013 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:50:02.013 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,756 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:50:02.013 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.014 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,756 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:50:02.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:02.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:50:02.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.436550140s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:50:02.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.7 2022-03-29T17:50:02.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:50:02.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:02.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.016 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 11 pg[2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:02.017 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,756 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:02.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 11 pg[2.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:02.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:50:02.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:02.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:50:02.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 8.365669250s + 0.000000000s) 2022-03-29T17:50:02.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,756 [shard 0] osd - pg_advance_map(id=9, detail=PGAdvanceMap(pg=2.6 from=11 to=12 do_init)): start 2022-03-29T17:50:02.019 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000396 0 0.000000 2022-03-29T17:50:02.019 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - Exiting state: Initial, entered at 1648576201.7566164, 0.0 spent on 0 events 2022-03-29T17:50:02.019 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:02.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.021 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000132 1 2022-03-29T17:50:01.757190+0000 2022-03-29T17:50:02.021 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - Exiting state: Reset, entered at 1648576201.7570746, 1648576201.75719 spent on 1 events 2022-03-29T17:50:02.021 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:02.022 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - Entering state: Started 2022-03-29T17:50:02.022 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:02.022 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - Entering state: Start 2022-03-29T17:50:02.022 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:50:02.023 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:50:02.023 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - Exiting state: Start, entered at 1648576201.7572873, 0.0 spent on 0 events 2022-03-29T17:50:02.023 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:50:02.023 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:50:02.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:02.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 11 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:02.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:50:02.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:50:02.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:02.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - Created 2.6 2022-03-29T17:50:02.026 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,757 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:50:02.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.439384460s int 16.000000000s) +create_info)): pg present 2022-03-29T17:50:02.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.439384460s int 16.000000000s) +create_info for pg: 2.6 2022-03-29T17:50:02.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,757 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:50:02.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 24.439384460s int 16.000000000s) 2022-03-29T17:50:02.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 24.366920471s 2022-03-29T17:50:02.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 24.366920471s 2022-03-29T17:50:02.028 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.000754 3 0.000244 2022-03-29T17:50:02.029 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,758 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576201.7573743, 0.000244428 spent on 3 events 2022-03-29T17:50:02.029 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:50:02.029 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,758 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:50:02.029 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.030 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,758 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.030 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:50:02.030 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:02.030 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:50:02.031 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.031 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.439384460s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:50:02.031 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.031 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - pg_advance_map(id=10, detail=PGAdvanceMap(pg=1.0 from=11 to=12)): complete 2022-03-29T17:50:02.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,758 [shard 0] osd - pg_advance_map(id=11, detail=PGAdvanceMap(pg=2.3 from=11 to=12)): complete 2022-03-29T17:50:02.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - pg_advance_map(id=15, detail=PGAdvanceMap(pg=2.7 from=11 to=12)): complete 2022-03-29T17:50:02.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.5 2022-03-29T17:50:02.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 12 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 11 2022-03-29T17:50:02.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - pg_advance_map(id=14, detail=PGAdvanceMap(pg=2.6 from=11 to=12)): complete 2022-03-29T17:50:02.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.4 2022-03-29T17:50:02.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 12 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 11 2022-03-29T17:50:02.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.039 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,759 [shard 0] osd - osd.1: now active 2022-03-29T17:50:02.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.6 2022-03-29T17:50:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 12 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 11 2022-03-29T17:50:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.3 2022-03-29T17:50:02.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,759 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 12 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 11 2022-03-29T17:50:02.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.7 2022-03-29T17:50:02.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 12 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 11 2022-03-29T17:50:02.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.353706360s int 16.000000000s) +create_info)): got map 12 2022-03-29T17:50:02.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:02.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.358369827s int 16.000000000s) +create_info)): start 2022-03-29T17:50:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,760 [shard 0] osd - compound_peering_request(id=3, detail=pg_create2(e12 {2.3=11,2022-03-29T17:50:00.701078+0000,2.4=11,2022-03-29T17:50:00.701078+0000,2.5=11,2022-03-29T17:50:00.701078+0000,2.6=11,2022-03-29T17:50:00.701078+0000,2.7=11,2022-03-29T17:50:00.701078+0000}) v2): starting 2022-03-29T17:50:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - handle_pg_create: 2.3 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=? pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): start 2022-03-29T17:50:02.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:02.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:02.047 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:02.047 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=0 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:02.047 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:02.047 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.436550140s int 16.000000000s) +create_info)): complete 2022-03-29T17:50:02.048 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.6 2022-03-29T17:50:02.048 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:50:02.048 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.048 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:02.049 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.049 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.7 2022-03-29T17:50:02.049 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: react 12 telling primary 2022-03-29T17:50:02.049 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.050 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.050 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.050 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,760 [shard 0] osd - handle_pg_create: 2.4 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=? pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): start 2022-03-29T17:50:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - handle_pg_create: 2.5 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=? pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): start 2022-03-29T17:50:02.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - handle_pg_create: 2.6 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=? pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): start 2022-03-29T17:50:02.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,761 [shard 0] osd - handle_pg_create: 2.7 e11 @1648576200.7010787 history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 pi ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,762 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=? pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): start 2022-03-29T17:50:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,762 [shard 0] osd - compound_peering_request(id=3, detail=pg_create2(e12 {2.3=11,2022-03-29T17:50:00.701078+0000,2.4=11,2022-03-29T17:50:00.701078+0000,2.5=11,2022-03-29T17:50:00.701078+0000,2.6=11,2022-03-29T17:50:00.701078+0000,2.7=11,2022-03-29T17:50:00.701078+0000}) v2): about to fork future 2022-03-29T17:50:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,762 [shard 0] osd - Creating 2.0 2022-03-29T17:50:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,762 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,762 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:02.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:02.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - pg_advance_map(id=9, detail=PGAdvanceMap(pg=2.6 from=11 to=12 do_init)): complete 2022-03-29T17:50:02.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:02.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.439384460s int 16.000000000s) +create_info)): complete 2022-03-29T17:50:02.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.6 2022-03-29T17:50:02.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: react 12 telling primary 2022-03-29T17:50:02.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,763 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:50:02.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,764 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,764 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,764 [shard 0] osd - pg_epoch 11 pg[2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:02.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,764 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:02.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,764 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=? pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): got map 12 2022-03-29T17:50:02.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,764 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 1} acting {2, 1} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:02.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:50:02.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:02.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:50:02.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 8.461044312s + 0.000000000s) 2022-03-29T17:50:02.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_advance_map(id=16, detail=PGAdvanceMap(pg=2.0 from=11 to=12 do_init)): start 2022-03-29T17:50:02.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=? pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): pg present 2022-03-29T17:50:02.068 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000803 0 0.000000 2022-03-29T17:50:02.068 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - Exiting state: Initial, entered at 1648576201.7646925, 0.0 spent on 0 events 2022-03-29T17:50:02.068 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:02.069 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:02.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:02.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:02.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:02.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:02.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.070 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000117 1 2022-03-29T17:50:01.765647+0000 2022-03-29T17:50:02.070 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - Exiting state: Reset, entered at 1648576201.765562, 1648576201.7656476 spent on 1 events 2022-03-29T17:50:02.071 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:02.071 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - Entering state: Started 2022-03-29T17:50:02.071 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:02.071 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - Entering state: Start 2022-03-29T17:50:02.072 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:50:02.072 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:50:02.072 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - Exiting state: Start, entered at 1648576201.7657373, 0.0 spent on 0 events 2022-03-29T17:50:02.072 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:50:02.073 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,765 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:50:02.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:02.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,765 [shard 0] osd - pg_epoch 11 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:50:02.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:50:02.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:50:02.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - OpSequencer::finish_op_in_order: op=125, last_started=125, last_unblocked=125, last_completed=124 2022-03-29T17:50:02.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=? pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): got map 12 2022-03-29T17:50:02.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:02.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - Created 2.0 2022-03-29T17:50:02.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,766 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:50:02.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.077 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=0 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:02.077 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.077 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - client_request(id=125, detail=m=[osd_op(client.4125.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {call lock.lock in=106b, cmpxattr striper.excl op 1 mode 1 in=12b, setxattr striper.excl (26) in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,766 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.353706360s int 16.000000000s) +create_info)): pg present 2022-03-29T17:50:02.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,767 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=? pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): got map 12 2022-03-29T17:50:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,767 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,767 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.330034256s int 16.000000000s) +create_info)): start 2022-03-29T17:50:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,767 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=? pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): pg present 2022-03-29T17:50:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,767 [shard 0] osd - handle_osd_map osd_map(11..12 src has 1..12) v4 2022-03-29T17:50:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,767 [shard 0] osd - handle_osd_map epochs [11..12], i have 12, src has [1..12] 2022-03-29T17:50:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,767 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,767 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 NullEvt +create_info for pg: 2.7 2022-03-29T17:50:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=? pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=? pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): pg present 2022-03-29T17:50:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.358369827s int 16.000000000s) +create_info)): got map 12 2022-03-29T17:50:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.353706360s int 16.000000000s) +create_info for pg: 2.0 2022-03-29T17:50:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:50:02.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 24.353706360s int 16.000000000s) 2022-03-29T17:50:02.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 24.464561462s 2022-03-29T17:50:02.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 24.464561462s 2022-03-29T17:50:02.167 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.002911 3 0.000218 2022-03-29T17:50:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,768 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576201.7658076, 0.000218768 spent on 3 events 2022-03-29T17:50:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:50:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,768 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:50:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,768 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:50:02.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:02.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:50:02.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,768 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.353706360s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:50:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=? pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): got map 12 2022-03-29T17:50:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - Creating 2.1 2022-03-29T17:50:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 NullEvt +create_info for pg: 2.5 2022-03-29T17:50:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=? pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:02.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=? pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): pg present 2022-03-29T17:50:02.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=? pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): got map 12 2022-03-29T17:50:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 NullEvt +create_info for pg: 2.6 2022-03-29T17:50:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,769 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=? pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.174 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:02.174 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:02.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 1} acting {2, 1} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:02.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:50:02.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:02.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:50:02.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 8.466328621s + 0.000000000s) 2022-03-29T17:50:02.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_advance_map(id=17, detail=PGAdvanceMap(pg=2.1 from=11 to=12 do_init)): start 2022-03-29T17:50:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=? pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): pg present 2022-03-29T17:50:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.000565 0 0.000000 2022-03-29T17:50:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - Exiting state: Initial, entered at 1648576201.7701287, 0.0 spent on 0 events 2022-03-29T17:50:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:02.178 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:02.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:02.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:02.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000112 1 2022-03-29T17:50:01.770818+0000 2022-03-29T17:50:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - Exiting state: Reset, entered at 1648576201.7707362, 1648576201.7708185 spent on 1 events 2022-03-29T17:50:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - Entering state: Started 2022-03-29T17:50:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - Entering state: Start 2022-03-29T17:50:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:50:02.181 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000036 0 0.000000 2022-03-29T17:50:02.181 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - Exiting state: Start, entered at 1648576201.7709072, 0.0 spent on 0 events 2022-03-29T17:50:02.181 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:50:02.181 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,770 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:50:02.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:02.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - client_request(id=126, detail=m=[osd_op(client.4125.0:125 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - client_request(id=126, detail=m=[osd_op(client.4125.0:125 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - pg_epoch 11 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:50:02.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:50:02.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:50:02.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.330034256s int 16.000000000s) +create_info)): got map 12 2022-03-29T17:50:02.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:02.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - Created 2.1 2022-03-29T17:50:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,771 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:50:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.0 2022-03-29T17:50:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:50:02.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:02.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:02.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.438203812s))): start 2022-03-29T17:50:02.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,771 [shard 0] osd - Creating 2.2 2022-03-29T17:50:02.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.358369827s int 16.000000000s) +create_info)): pg present 2022-03-29T17:50:02.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 NullEvt +create_info for pg: 2.3 2022-03-29T17:50:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=? pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=? pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): complete 2022-03-29T17:50:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 NullEvt +create_info for pg: 2.4 2022-03-29T17:50:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=? pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:50:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,772 [shard 0] osd - pg_epoch 11 pg[2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:50:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,772 [shard 0] osd - Entering state: Initial 2022-03-29T17:50:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - pg_advance_map(id=16, detail=PGAdvanceMap(pg=2.0 from=11 to=12 do_init)): complete 2022-03-29T17:50:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - client_request(id=126, detail=m=[osd_op(client.4125.0:125 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,772 [shard 0] osd - OpSequencer::start_op: op=126, last_started=125, last_unblocked=125, last_completed=125 2022-03-29T17:50:02.191 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,772 [shard 0] osd - compound_peering_request(id=3, detail=pg_create2(e12 {2.3=11,2022-03-29T17:50:00.701078+0000,2.4=11,2022-03-29T17:50:00.701078+0000,2.5=11,2022-03-29T17:50:00.701078+0000,2.6=11,2022-03-29T17:50:00.701078+0000,2.7=11,2022-03-29T17:50:00.701078+0000}) v2): sub events complete 2022-03-29T17:50:02.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 11 pg[2.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:50:02.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:50:02.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:50:02.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:50:02.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:50:02.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 8.469394684s + 0.000000000s) 2022-03-29T17:50:02.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_advance_map(id=18, detail=PGAdvanceMap(pg=2.2 from=11 to=12 do_init)): start 2022-03-29T17:50:02.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.438203812s))): got map 12 2022-03-29T17:50:02.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.358369827s int 16.000000000s) +create_info for pg: 2.1 2022-03-29T17:50:02.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:50:02.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 24.358369827s int 16.000000000s) 2022-03-29T17:50:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 24.469774246s 2022-03-29T17:50:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 24.469774246s 2022-03-29T17:50:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.002935 3 0.000198 2022-03-29T17:50:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,773 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576201.7709706, 0.00019899 spent on 3 events 2022-03-29T17:50:02.196 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,773 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:50:02.196 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,773 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:50:02.196 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.196 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,774 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:50:02.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:02.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:50:02.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.358369827s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:50:02.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:02.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:02.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.429607391s))): start 2022-03-29T17:50:02.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:02.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.438203812s))): pg present 2022-03-29T17:50:02.199 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,774 [shard 0] osd - compound_peering_request(id=3, detail=pg_create2(e12 {2.3=11,2022-03-29T17:50:00.701078+0000,2.4=11,2022-03-29T17:50:00.701078+0000,2.5=11,2022-03-29T17:50:00.701078+0000,2.6=11,2022-03-29T17:50:00.701078+0000,2.7=11,2022-03-29T17:50:00.701078+0000}) v2): complete 2022-03-29T17:50:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002023 0 0.000000 2022-03-29T17:50:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,774 [shard 0] osd - Exiting state: Initial, entered at 1648576201.7726521, 0.0 spent on 0 events 2022-03-29T17:50:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:50:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,774 [shard 0] osd - Entering state: Reset 2022-03-29T17:50:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 11 2022-03-29T17:50:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:50:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:50:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:50:02.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,774 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.202 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Reset 0.000107 1 2022-03-29T17:50:01.774808+0000 2022-03-29T17:50:02.202 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - Exiting state: Reset, entered at 1648576201.774717, 1648576201.7748082 spent on 1 events 2022-03-29T17:50:02.202 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:50:02.203 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - Entering state: Started 2022-03-29T17:50:02.203 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:50:02.203 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - Entering state: Start 2022-03-29T17:50:02.204 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:50:02.204 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Start 0.000073 0 0.000000 2022-03-29T17:50:02.204 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - Exiting state: Start, entered at 1648576201.7752388, 0.0 spent on 0 events 2022-03-29T17:50:02.204 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:50:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:50:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 11 2022-03-29T17:50:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 11 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:50:02.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:50:02.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:50:02.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:50:02.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 11 while current is 12 2022-03-29T17:50:02.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - Created 2.2 2022-03-29T17:50:02.207 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,775 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:50:02.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,775 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.353706360s int 16.000000000s) +create_info)): complete 2022-03-29T17:50:02.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=? pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): complete 2022-03-29T17:50:02.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.330034256s int 16.000000000s) +create_info)): pg present 2022-03-29T17:50:02.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=? pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): complete 2022-03-29T17:50:02.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.429607391s))): got map 12 2022-03-29T17:50:02.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.438203812s) for pg: 2.3 2022-03-29T17:50:02.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 24.438203812s 2022-03-29T17:50:02.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.438203812s (sent 24.438203812s) 2022-03-29T17:50:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:50:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:50:02.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:4 2022-03-29T17:50:02.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:50:02.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 24.438203812s 2022-03-29T17:50:02.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.438203812s (sent 24.472705841s) 2022-03-29T17:50:02.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,776 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:50:02.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active state: mnow 8.473089218s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:50:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - send_pg_created 2022-03-29T17:50:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:50:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:50:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:5 2022-03-29T17:50:02.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:50:02.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:50:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): start 2022-03-29T17:50:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:50:02.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,777 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:6 2022-03-29T17:50:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.438203812s))): submitting ctx 2022-03-29T17:50:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:02.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 11 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.330034256s int 16.000000000s) +create_info for pg: 2.2 2022-03-29T17:50:02.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:50:02.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:50:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 24.330034256s int 16.000000000s) 2022-03-29T17:50:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 24.474807739s 2022-03-29T17:50:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,778 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 24.474807739s 2022-03-29T17:50:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,779 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.003667 3 0.000601 2022-03-29T17:50:02.226 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,779 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576201.7753556, 0.000601917 spent on 3 events 2022-03-29T17:50:02.226 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,779 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:50:02.226 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,779 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:50:02.226 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,779 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.227 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,779 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:50:02.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=0/0 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:50:02.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:50:02.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:50:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.330034256s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:50:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.0 2022-03-29T17:50:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: react 12 telling primary 2022-03-29T17:50:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:50:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.429607391s))): pg present 2022-03-29T17:50:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,779 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=? pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): complete 2022-03-29T17:50:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): pg present 2022-03-29T17:50:02.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - pg_advance_map(id=17, detail=PGAdvanceMap(pg=2.1 from=11 to=12 do_init)): complete 2022-03-29T17:50:02.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): start 2022-03-29T17:50:02.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:02.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=? pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 NullEvt +create_info)): complete 2022-03-29T17:50:02.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): pg present 2022-03-29T17:50:02.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - client_request(id=126, detail=m=[osd_op(client.4125.0:125 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.358369827s int 16.000000000s) +create_info)): complete 2022-03-29T17:50:02.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:02.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,780 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.429607391s) for pg: 2.4 2022-03-29T17:50:02.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 24.429607391s 2022-03-29T17:50:02.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.429607391s (sent 24.429607391s) 2022-03-29T17:50:02.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:50:02.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:4 2022-03-29T17:50:02.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:50:02.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 24.429607391s 2022-03-29T17:50:02.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.429607391s (sent 24.477348328s) 2022-03-29T17:50:02.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active state: mnow 8.477704048s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:50:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - send_pg_created 2022-03-29T17:50:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:50:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,781 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:5 2022-03-29T17:50:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:50:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:50:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): start 2022-03-29T17:50:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:6 2022-03-29T17:50:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.429607391s))): submitting ctx 2022-03-29T17:50:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:02.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,782 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:02.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_advance_map(id=18, detail=PGAdvanceMap(pg=2.2 from=11 to=12 do_init)): complete 2022-03-29T17:50:02.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered for pg: 2.3 2022-03-29T17:50:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.040442 4 0.002624 2022-03-29T17:50:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,783 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576201.742853, 0.002624918 spent on 4 events 2022-03-29T17:50:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:50:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,783 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:50:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:50:02.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:7 2022-03-29T17:50:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000369 0 0.000000 2022-03-29T17:50:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,783 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576201.7833736, 0.0 spent on 0 events 2022-03-29T17:50:02.254 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:50:02.254 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,783 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:50:02.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:50:02.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,783 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:8 2022-03-29T17:50:02.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): submitting ctx 2022-03-29T17:50:02.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.438203812s))): complete 2022-03-29T17:50:02.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.1 2022-03-29T17:50:02.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:50:02.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:02.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): start 2022-03-29T17:50:02.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): pg present 2022-03-29T17:50:02.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): pg present 2022-03-29T17:50:02.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=11 evt=epoch_sent: 12 epoch_requested: 11 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 24.330034256s int 16.000000000s) +create_info)): complete 2022-03-29T17:50:02.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 IntervalFlush for pg: 2.2 2022-03-29T17:50:02.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:50:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): submitting ctx 2022-03-29T17:50:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,784 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.1 2022-03-29T17:50:02.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: react 12 telling primary 2022-03-29T17:50:02.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:50:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.size 2022-03-29T17:50:02.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12 for pg: 2.2 2022-03-29T17:50:02.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 unknown state: react 12 telling primary 2022-03-29T17:50:02.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): submitting ctx 2022-03-29T17:50:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:50:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,785 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered for pg: 2.4 2022-03-29T17:50:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,785 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.051685 4 0.002607 2022-03-29T17:50:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,786 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576201.7342825, 0.002607661 spent on 4 events 2022-03-29T17:50:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:50:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,786 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:50:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:7 2022-03-29T17:50:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000417 0 0.000000 2022-03-29T17:50:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,786 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576201.7860298, 0.0 spent on 0 events 2022-03-29T17:50:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:50:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,786 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:50:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:50:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:8 2022-03-29T17:50:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): submitting ctx 2022-03-29T17:50:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,786 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.allocated 2022-03-29T17:50:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,787 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op getxattr 2022-03-29T17:50:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,787 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,787 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.version 2022-03-29T17:50:02.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,787 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.429607391s))): complete 2022-03-29T17:50:02.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,787 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:02.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,787 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:125 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:02.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,787 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,787 [shard 0] osd - OpSequencer::finish_op_in_order: op=126, last_started=126, last_unblocked=126, last_completed=125 2022-03-29T17:50:02.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,788 [shard 0] osd - client_request(id=126, detail=m=[osd_op(client.4125.0:125 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {getxattr striper.excl (26) in=12b out=26b, getxattr striper.size (16) in=12b out=16b, getxattr striper.allocated (16) in=17b out=16b, getxattr striper.version (16) in=15b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,788 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,788 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.431535721s))): start 2022-03-29T17:50:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,788 [shard 0] osd - client_request(id=127, detail=m=[osd_op(client.4125.0:126 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,788 [shard 0] osd - client_request(id=127, detail=m=[osd_op(client.4125.0:126 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,788 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.431535721s))): got map 12 2022-03-29T17:50:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,788 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.431535721s))): pg present 2022-03-29T17:50:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,788 [shard 0] osd - client_request(id=127, detail=m=[osd_op(client.4125.0:126 1.0 1.756970d8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - OpSequencer::start_op: op=127, last_started=126, last_unblocked=126, last_completed=126 2022-03-29T17:50:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.431535721s) for pg: 2.5 2022-03-29T17:50:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 24.431535721s 2022-03-29T17:50:02.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.431535721s (sent 24.431535721s) 2022-03-29T17:50:02.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:50:02.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:4 2022-03-29T17:50:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:50:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 24.431535721s 2022-03-29T17:50:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.431535721s (sent 24.485576630s) 2022-03-29T17:50:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,789 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active state: mnow 8.485930443s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:50:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - send_pg_created 2022-03-29T17:50:02.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:50:02.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:5 2022-03-29T17:50:02.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:50:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:50:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): start 2022-03-29T17:50:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:6 2022-03-29T17:50:02.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.431535721s))): submitting ctx 2022-03-29T17:50:02.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,790 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,791 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,791 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.436550140s))): start 2022-03-29T17:50:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,791 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,791 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,791 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 IntervalFlush)): complete 2022-03-29T17:50:02.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,791 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,791 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): pg present 2022-03-29T17:50:02.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,791 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.436550140s))): got map 12 2022-03-29T17:50:02.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 ActivateCommitted from 12 processed at 12)): complete 2022-03-29T17:50:02.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.436550140s))): pg present 2022-03-29T17:50:02.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): complete 2022-03-29T17:50:02.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): complete 2022-03-29T17:50:02.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - client_request(id=127, detail=m=[osd_op(client.4125.0:126 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.439384460s))): start 2022-03-29T17:50:02.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op read 2022-03-29T17:50:02.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - handling op read on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered for pg: 2.5 2022-03-29T17:50:02.292 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,792 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.056731 4 0.002367 2022-03-29T17:50:02.293 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,792 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576201.7360654, 0.002367174 spent on 4 events 2022-03-29T17:50:02.293 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,792 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:50:02.293 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,792 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:50:02.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,792 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:7 2022-03-29T17:50:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000363 0 0.000000 2022-03-29T17:50:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,793 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576201.7928674, 0.0 spent on 0 events 2022-03-29T17:50:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:50:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,793 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:50:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:50:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:8 2022-03-29T17:50:02.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): submitting ctx 2022-03-29T17:50:02.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.436550140s) for pg: 2.7 2022-03-29T17:50:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 24.436550140s 2022-03-29T17:50:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.436550140s (sent 24.436550140s) 2022-03-29T17:50:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:50:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,793 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:4 2022-03-29T17:50:02.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:50:02.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 24.436550140s 2022-03-29T17:50:02.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.436550140s (sent 24.490119934s) 2022-03-29T17:50:02.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active state: mnow 8.490500450s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:50:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - send_pg_created 2022-03-29T17:50:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:50:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,794 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:5 2022-03-29T17:50:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:50:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:50:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): start 2022-03-29T17:50:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:6 2022-03-29T17:50:02.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.436550140s))): submitting ctx 2022-03-29T17:50:02.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - read: data length: 1 2022-03-29T17:50:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,795 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:126 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.439384460s))): got map 12 2022-03-29T17:50:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.431535721s))): complete 2022-03-29T17:50:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): pg present 2022-03-29T17:50:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.439384460s))): pg present 2022-03-29T17:50:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - OpSequencer::finish_op_in_order: op=127, last_started=127, last_unblocked=127, last_completed=126 2022-03-29T17:50:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - client_request(id=127, detail=m=[osd_op(client.4125.0:126 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {read 0~1 out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered for pg: 2.7 2022-03-29T17:50:02.314 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,796 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.055622 4 0.002762 2022-03-29T17:50:02.314 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,796 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576201.7410989, 0.002762727 spent on 4 events 2022-03-29T17:50:02.314 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,796 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:50:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,796 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:50:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,796 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:7 2022-03-29T17:50:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000430 0 0.000000 2022-03-29T17:50:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,797 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576201.796804, 0.0 spent on 0 events 2022-03-29T17:50:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:50:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,797 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:50:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:50:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:8 2022-03-29T17:50:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): submitting ctx 2022-03-29T17:50:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): complete 2022-03-29T17:50:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.439384460s) for pg: 2.6 2022-03-29T17:50:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 24.439384460s 2022-03-29T17:50:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,797 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.439384460s (sent 24.439384460s) 2022-03-29T17:50:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:50:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:4 2022-03-29T17:50:02.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:50:02.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 24.439384460s 2022-03-29T17:50:02.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.439384460s (sent 24.494297028s) 2022-03-29T17:50:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active state: mnow 8.494657516s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:50:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - send_pg_created 2022-03-29T17:50:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:50:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,798 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:5 2022-03-29T17:50:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:50:02.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:50:02.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): start 2022-03-29T17:50:02.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:6 2022-03-29T17:50:02.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.439384460s))): submitting ctx 2022-03-29T17:50:02.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,799 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.436550140s))): complete 2022-03-29T17:50:02.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - client_request(id=128, detail=m=[osd_op(client.4125.0:127 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - client_request(id=128, detail=m=[osd_op(client.4125.0:127 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): pg present 2022-03-29T17:50:02.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - client_request(id=128, detail=m=[osd_op(client.4125.0:127 1.0 1.4e660c0c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - OpSequencer::start_op: op=128, last_started=127, last_unblocked=127, last_completed=127 2022-03-29T17:50:02.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - client_request(id=128, detail=m=[osd_op(client.4125.0:127 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:02.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op read 2022-03-29T17:50:02.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - handling op read on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,800 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered for pg: 2.6 2022-03-29T17:50:02.335 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,800 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.057040 4 0.002448 2022-03-29T17:50:02.335 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,800 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576201.7439067, 0.002448479 spent on 4 events 2022-03-29T17:50:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:50:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,801 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:50:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:50:02.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:50:02.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:7 2022-03-29T17:50:02.338 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000219 0 0.000000 2022-03-29T17:50:02.339 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,801 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576201.8010085, 0.0 spent on 0 events 2022-03-29T17:50:02.339 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:50:02.339 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:01,801 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:50:02.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:50:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:8 2022-03-29T17:50:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): submitting ctx 2022-03-29T17:50:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - read: data length: 16 2022-03-29T17:50:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,801 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:127 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): complete 2022-03-29T17:50:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - OpSequencer::finish_op_in_order: op=128, last_started=128, last_unblocked=128, last_completed=127 2022-03-29T17:50:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - client_request(id=128, detail=m=[osd_op(client.4125.0:127 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {read 24~16 out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - client_request(id=129, detail=m=[osd_op(client.4125.0:128 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - client_request(id=129, detail=m=[osd_op(client.4125.0:128 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - client_request(id=129, detail=m=[osd_op(client.4125.0:128 1.0 1.9c5c9294 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - OpSequencer::start_op: op=129, last_started=128, last_unblocked=128, last_completed=128 2022-03-29T17:50:02.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - do_recover_missing check for recovery, 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - client_request(id=129, detail=m=[osd_op(client.4125.0:128 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - with_head_obc 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - with_head_obc: found 1:29493a39:devicehealth::main.db-wal.0000000000000000:head in cache 2022-03-29T17:50:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - do_osd_ops_execute: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head - handling op getxattr 2022-03-29T17:50:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - handling op getxattr on object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:02.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,802 [shard 0] osd - getxattr on obj=1:29493a39:devicehealth::main.db-wal.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:02.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,803 [shard 0] osd - rollback_obc_if_modified: object 1:29493a39:devicehealth::main.db-wal.0000000000000000:head got error generic:2, need_rollback=false 2022-03-29T17:50:02.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,803 [shard 0] osd - with_head_obc: released 1:29493a39:devicehealth::main.db-wal.0000000000000000:head 2022-03-29T17:50:02.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,803 [shard 0] osd - OpSequencer::finish_op_in_order: op=129, last_started=129, last_unblocked=129, last_completed=128 2022-03-29T17:50:02.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,803 [shard 0] osd - client_request(id=129, detail=m=[osd_op(client.4125.0:128 1.0 1:29493a39:devicehealth::main.db-wal.0000000000000000:head {getxattr striper.excl in=12b, getxattr striper.size in=12b, getxattr striper.allocated in=17b, getxattr striper.version in=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,803 [shard 0] osd - client_request(id=130, detail=m=[osd_op(client.4125.0:129 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,803 [shard 0] osd - client_request(id=130, detail=m=[osd_op(client.4125.0:129 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,803 [shard 0] osd - client_request(id=130, detail=m=[osd_op(client.4125.0:129 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,803 [shard 0] osd - OpSequencer::start_op: op=130, last_started=129, last_unblocked=129, last_completed=129 2022-03-29T17:50:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - client_request(id=130, detail=m=[osd_op(client.4125.0:129 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'86 mlcod 11'86 active+clean calc_trim_to_aggressive limit = 11'86 2022-03-29T17:50:02.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=11'86 mlcod 11'86 active+clean append_log log((0'0,11'86], crt=11'86) {12'87 (11'84) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:129 2022-03-29T17:50:01.803482+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=11'86 lcod 11'86 mlcod 11'86 active+clean add_log_entry 12'87 (11'84) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:129 2022-03-29T17:50:01.803482+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'87 lcod 11'86 mlcod 11'86 active+clean append_log approx pg log length = 87 2022-03-29T17:50:02.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'87 lcod 11'86 mlcod 11'86 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'87, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,804 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.439384460s))): complete 2022-03-29T17:50:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - client_request(id=131, detail=m=[osd_op(client.4125.0:130 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - client_request(id=131, detail=m=[osd_op(client.4125.0:130 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - client_request(id=131, detail=m=[osd_op(client.4125.0:130 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - OpSequencer::start_op: op=131, last_started=130, last_unblocked=130, last_completed=129 2022-03-29T17:50:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - client_request(id=131, detail=m=[osd_op(client.4125.0:130 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,805 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'87 lcod 11'86 mlcod 11'86 active+clean calc_trim_to_aggressive limit = 11'86 2022-03-29T17:50:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'87 lcod 11'86 mlcod 11'86 active+clean append_log log((0'0,12'87], crt=12'87) {12'88 (12'87) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:130 2022-03-29T17:50:01.803513+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'87 lcod 11'86 mlcod 11'86 active+clean add_log_entry 12'88 (12'87) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:130 2022-03-29T17:50:01.803513+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'88 lcod 11'86 mlcod 11'86 active+clean append_log approx pg log length = 88 2022-03-29T17:50:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'88 lcod 11'86 mlcod 11'86 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'88, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,806 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - client_request(id=132, detail=m=[osd_op(client.4125.0:131 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - client_request(id=132, detail=m=[osd_op(client.4125.0:131 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): complete 2022-03-29T17:50:02.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - client_request(id=132, detail=m=[osd_op(client.4125.0:131 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - OpSequencer::start_op: op=132, last_started=131, last_unblocked=131, last_completed=129 2022-03-29T17:50:02.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - client_request(id=132, detail=m=[osd_op(client.4125.0:131 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:02.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,807 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:02.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'88 lcod 11'86 mlcod 11'86 active+clean calc_trim_to_aggressive limit = 11'86 2022-03-29T17:50:02.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'88 lcod 11'86 mlcod 11'86 active+clean append_log log((0'0,12'88], crt=12'88) {12'89 (12'88) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:131 2022-03-29T17:50:01.803533+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'88 lcod 11'86 mlcod 11'86 active+clean add_log_entry 12'89 (12'88) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:131 2022-03-29T17:50:01.803533+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'89 lcod 11'86 mlcod 11'86 active+clean append_log approx pg log length = 89 2022-03-29T17:50:02.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'89 lcod 11'86 mlcod 11'86 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'89, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - client_request(id=133, detail=m=[osd_op(client.4125.0:132 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,808 [shard 0] osd - client_request(id=133, detail=m=[osd_op(client.4125.0:132 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.362 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,809 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:02.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,809 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): start 2022-03-29T17:50:02.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,809 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): got map 12 2022-03-29T17:50:02.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,809 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): pg present 2022-03-29T17:50:02.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,809 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:02.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,809 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): start 2022-03-29T17:50:02.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:02.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s) 2022-03-29T17:50:02.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 24.418985367s 2022-03-29T17:50:02.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 24.418985367s 2022-03-29T17:50:02.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): got map 12 2022-03-29T17:50:02.366 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): pg present 2022-03-29T17:50:02.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - client_request(id=133, detail=m=[osd_op(client.4125.0:132 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,810 [shard 0] osd - OpSequencer::start_op: op=133, last_started=132, last_unblocked=132, last_completed=129 2022-03-29T17:50:02.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - client_request(id=133, detail=m=[osd_op(client.4125.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:02.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:02.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'89 lcod 11'86 mlcod 11'86 active+clean calc_trim_to_aggressive limit = 11'86 2022-03-29T17:50:02.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'89 lcod 11'86 mlcod 11'86 active+clean append_log log((0'0,12'89], crt=12'89) {12'90 (12'89) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:132 2022-03-29T17:50:01.803548+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'89 lcod 11'86 mlcod 11'86 active+clean add_log_entry 12'90 (12'89) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:132 2022-03-29T17:50:01.803548+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'90 lcod 11'86 mlcod 11'86 active+clean append_log approx pg log length = 90 2022-03-29T17:50:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'90 lcod 11'86 mlcod 11'86 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'90, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,811 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,812 [shard 0] osd - client_request(id=134, detail=m=[osd_op(client.4125.0:133 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,812 [shard 0] osd - client_request(id=134, detail=m=[osd_op(client.4125.0:133 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,812 [shard 0] osd - client_request(id=134, detail=m=[osd_op(client.4125.0:133 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,812 [shard 0] osd - OpSequencer::start_op: op=134, last_started=133, last_unblocked=133, last_completed=129 2022-03-29T17:50:02.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,812 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,813 [shard 0] osd - client_request(id=134, detail=m=[osd_op(client.4125.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,813 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,813 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,813 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:02.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,813 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): start 2022-03-29T17:50:02.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,813 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): got map 12 2022-03-29T17:50:02.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,813 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): pg present 2022-03-29T17:50:02.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:02.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): start 2022-03-29T17:50:02.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:02.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s) 2022-03-29T17:50:02.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 24.447330475s 2022-03-29T17:50:02.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 24.447330475s 2022-03-29T17:50:02.398 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.398 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.398 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.398 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): got map 12 2022-03-29T17:50:02.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): pg present 2022-03-29T17:50:02.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:02.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,814 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.353706360s))): start 2022-03-29T17:50:02.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:02.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11 2022-03-29T17:50:02.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - peering_event(id=13, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): complete 2022-03-29T17:50:02.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.353706360s))): got map 12 2022-03-29T17:50:02.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.353706360s))): pg present 2022-03-29T17:50:02.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:02.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): start 2022-03-29T17:50:02.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.353706360s) for pg: 2.0 2022-03-29T17:50:02.402 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 24.353706360s 2022-03-29T17:50:02.402 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.353706360s (sent 24.353706360s) 2022-03-29T17:50:02.402 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,815 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:50:02.402 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:4 2022-03-29T17:50:02.405 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:50:02.405 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 24.353706360s 2022-03-29T17:50:02.405 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.353706360s (sent 24.449274063s) 2022-03-29T17:50:02.405 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active state: mnow 8.449521065s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:50:02.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - send_pg_created 2022-03-29T17:50:02.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:50:02.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:5 2022-03-29T17:50:02.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:50:02.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:50:02.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): start 2022-03-29T17:50:02.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.412 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.412 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.412 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.412 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,816 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.413 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.413 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.413 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:6 2022-03-29T17:50:02.413 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.353706360s))): submitting ctx 2022-03-29T17:50:02.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - peering_event(id=14, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): complete 2022-03-29T17:50:02.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): got map 12 2022-03-29T17:50:02.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): pg present 2022-03-29T17:50:02.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:02.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.358369827s))): start 2022-03-29T17:50:02.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,817 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): pg present 2022-03-29T17:50:02.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:02.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11 2022-03-29T17:50:02.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:50:02.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.358369827s))): got map 12 2022-03-29T17:50:02.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.358369827s))): pg present 2022-03-29T17:50:02.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - peering_event(id=15, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.353706360s))): complete 2022-03-29T17:50:02.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered for pg: 2.0 2022-03-29T17:50:02.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.095130 3 0.005135 2022-03-29T17:50:02.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,818 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576201.7234344, 0.005135113 spent on 3 events 2022-03-29T17:50:02.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:50:02.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,818 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:50:02.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,818 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.421 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.421 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:7 2022-03-29T17:50:02.421 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000452 0 0.000000 2022-03-29T17:50:02.421 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,819 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576201.8186452, 0.0 spent on 0 events 2022-03-29T17:50:02.421 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:50:02.422 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,819 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:50:02.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:50:02.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:8 2022-03-29T17:50:02.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): submitting ctx 2022-03-29T17:50:02.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - peering_event(id=16, detail=PeeringEvent(from=1 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.438203812s ub 24.472705841s int 16.000000000s))): complete 2022-03-29T17:50:02.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.358369827s) for pg: 2.1 2022-03-29T17:50:02.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 24.358369827s 2022-03-29T17:50:02.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.358369827s (sent 24.358369827s) 2022-03-29T17:50:02.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:50:02.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.425 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.425 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,819 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.425 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.425 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:4 2022-03-29T17:50:02.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:50:02.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 24.358369827s 2022-03-29T17:50:02.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.358369827s (sent 24.453264236s) 2022-03-29T17:50:02.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active state: mnow 8.453654289s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:50:02.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - send_pg_created 2022-03-29T17:50:02.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:50:02.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,820 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:5 2022-03-29T17:50:02.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:50:02.433 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:50:02.433 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): start 2022-03-29T17:50:02.433 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.433 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.433 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:6 2022-03-29T17:50:02.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.358369827s))): submitting ctx 2022-03-29T17:50:02.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,821 [shard 0] osd - peering_event(id=17, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): complete 2022-03-29T17:50:02.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - peering_event(id=18, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.358369827s))): complete 2022-03-29T17:50:02.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): pg present 2022-03-29T17:50:02.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered for pg: 2.1 2022-03-29T17:50:02.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.094991 3 0.004746 2022-03-29T17:50:02.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,822 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576201.727279, 0.004746456 spent on 3 events 2022-03-29T17:50:02.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:50:02.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,822 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:50:02.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:50:02.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:50:02.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:50:02.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:7 2022-03-29T17:50:02.440 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000378 0 0.000000 2022-03-29T17:50:02.440 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,822 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576201.8223443, 0.0 spent on 0 events 2022-03-29T17:50:02.441 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:50:02.441 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:01,822 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:50:02.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:50:02.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,822 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:8 2022-03-29T17:50:02.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): submitting ctx 2022-03-29T17:50:02.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,813 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:02.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:02.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'90 lcod 11'86 mlcod 11'86 active+clean calc_trim_to_aggressive limit = 11'86 2022-03-29T17:50:02.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'90 lcod 11'86 mlcod 11'86 active+clean append_log log((0'0,12'90], crt=12'90) {12'91 (12'90) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:133 2022-03-29T17:50:01.803591+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'90 lcod 11'86 mlcod 11'86 active+clean add_log_entry 12'91 (12'90) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:133 2022-03-29T17:50:01.803591+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'91 lcod 11'86 mlcod 11'86 active+clean append_log approx pg log length = 91 2022-03-29T17:50:02.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'91 lcod 11'86 mlcod 11'86 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,823 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.450 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,824 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'91, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.450 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,824 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,824 [shard 0] osd - peering_event(id=19, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): complete 2022-03-29T17:50:02.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,824 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:02.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,824 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.477348328s))): start 2022-03-29T17:50:02.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,825 [shard 0] osd - client_request(id=135, detail=m=[osd_op(client.4125.0:134 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,825 [shard 0] osd - client_request(id=135, detail=m=[osd_op(client.4125.0:134 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,825 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.477348328s))): got map 12 2022-03-29T17:50:02.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,825 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.477348328s))): pg present 2022-03-29T17:50:02.477 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:02.478 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.330034256s))): start 2022-03-29T17:50:02.478 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): complete 2022-03-29T17:50:02.478 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.330034256s))): got map 12 2022-03-29T17:50:02.479 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:02.479 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11 2022-03-29T17:50:02.479 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.480 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.480 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.480 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.330034256s))): pg present 2022-03-29T17:50:02.480 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.330034256s) for pg: 2.2 2022-03-29T17:50:02.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 24.330034256s 2022-03-29T17:50:02.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.330034256s (sent 24.330034256s) 2022-03-29T17:50:02.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:50:02.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,826 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.482 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:50:02.482 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.482 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.482 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.484 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 12:4 2022-03-29T17:50:02.484 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:50:02.484 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 24.330034256s 2022-03-29T17:50:02.484 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 24.330034256s (sent 24.436038971s) 2022-03-29T17:50:02.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:50:02.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:50:02.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:50:02.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:50:02.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active state: mnow 8.436342239s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:50:02.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - send_pg_created 2022-03-29T17:50:02.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:50:02.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:50:02.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:5 2022-03-29T17:50:02.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:50:02.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:50:02.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): start 2022-03-29T17:50:02.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:50:02.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.492 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.492 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.492 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.492 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,827 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:6 2022-03-29T17:50:02.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.330034256s))): submitting ctx 2022-03-29T17:50:02.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:02.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): start 2022-03-29T17:50:02.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): got map 12 2022-03-29T17:50:02.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): complete 2022-03-29T17:50:02.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): pg present 2022-03-29T17:50:02.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): pg present 2022-03-29T17:50:02.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,828 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease_ack(ruub 24.330034256s))): complete 2022-03-29T17:50:02.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:02.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11 2022-03-29T17:50:02.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:50:02.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered for pg: 2.2 2022-03-29T17:50:02.497 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.106202 4 0.005422 2022-03-29T17:50:02.497 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,829 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576201.7232313, 0.005422071 spent on 4 events 2022-03-29T17:50:02.498 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:50:02.498 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,829 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:50:02.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:50:02.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:50:02.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:50:02.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:50:02.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:50:02.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:50:02.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:50:02.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:50:02.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:50:02.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.501 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active publish_stats_to_osd 12:7 2022-03-29T17:50:02.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000374 0 0.000000 2022-03-29T17:50:02.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,829 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576201.8295023, 0.0 spent on 0 events 2022-03-29T17:50:02.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:50:02.502 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:01,829 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:50:02.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,829 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:50:02.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:8 2022-03-29T17:50:02.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): submitting ctx 2022-03-29T17:50:02.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=1 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.429607391s ub 24.477348328s int 16.000000000s))): complete 2022-03-29T17:50:02.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:02.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): start 2022-03-29T17:50:02.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): got map 12 2022-03-29T17:50:02.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 AllReplicasRecovered)): complete 2022-03-29T17:50:02.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,830 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): pg present 2022-03-29T17:50:02.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:02.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s) 2022-03-29T17:50:02.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 24.439996719s 2022-03-29T17:50:02.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 24.439996719s 2022-03-29T17:50:02.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:02.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): start 2022-03-29T17:50:02.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): got map 12 2022-03-29T17:50:02.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): complete 2022-03-29T17:50:02.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,831 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): pg present 2022-03-29T17:50:02.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:02.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11 2022-03-29T17:50:02.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:02.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): start 2022-03-29T17:50:02.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): got map 12 2022-03-29T17:50:02.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): complete 2022-03-29T17:50:02.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): pg present 2022-03-29T17:50:02.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,832 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:02.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11 2022-03-29T17:50:02.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:50:02.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:02.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): start 2022-03-29T17:50:02.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): got map 12 2022-03-29T17:50:02.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): pg present 2022-03-29T17:50:02.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.431535721s ub 24.485576630s int 16.000000000s))): complete 2022-03-29T17:50:02.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:02.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,833 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): start 2022-03-29T17:50:02.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:02.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s) 2022-03-29T17:50:02.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 24.443010330s 2022-03-29T17:50:02.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 24.443010330s 2022-03-29T17:50:02.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): got map 12 2022-03-29T17:50:02.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): pg present 2022-03-29T17:50:02.516 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:02.516 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11 2022-03-29T17:50:02.516 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,834 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.516 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:02.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): start 2022-03-29T17:50:02.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): complete 2022-03-29T17:50:02.518 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): got map 12 2022-03-29T17:50:02.518 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): pg present 2022-03-29T17:50:02.518 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): complete 2022-03-29T17:50:02.518 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:02.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,835 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): start 2022-03-29T17:50:02.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:02.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11 2022-03-29T17:50:02.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:50:02.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): got map 12 2022-03-29T17:50:02.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): pg present 2022-03-29T17:50:02.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:02.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): start 2022-03-29T17:50:02.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:02.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s) 2022-03-29T17:50:02.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,836 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 24.445840836s 2022-03-29T17:50:02.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 24.445840836s 2022-03-29T17:50:02.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.523 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.523 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=1 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.436550140s ub 24.490119934s int 16.000000000s))): complete 2022-03-29T17:50:02.523 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): got map 12 2022-03-29T17:50:02.523 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): pg present 2022-03-29T17:50:02.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:02.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): start 2022-03-29T17:50:02.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,837 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:02.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11 2022-03-29T17:50:02.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.1 pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): complete 2022-03-29T17:50:02.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): got map 12 2022-03-29T17:50:02.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): pg present 2022-03-29T17:50:02.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - replicated_request(id=87, detail=RepRequest(from=1 req=osd_repop(client.4125.0:129 1.0 e12/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 12'87, mlcod=12'87) v3)) start 2022-03-29T17:50:02.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:02.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11 2022-03-29T17:50:02.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:50:02.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,838 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): complete 2022-03-29T17:50:02.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - pg_epoch 12 pg[1.0( v 11'86 (0'0,11'86] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 mlcod 11'86 active append_log log((0'0,11'86], crt=11'86) {12'87 (11'84) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:129 2022-03-29T17:50:01.803482+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=11'86 lcod 11'86 mlcod 11'86 active add_log_entry 12'87 (11'84) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:129 2022-03-29T17:50:01.803482+0000 0 ObjectCleanRegions clean_offsets: [512~18446744073709551103], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'87 lcod 11'86 mlcod 11'86 active append_log approx pg log length = 87 2022-03-29T17:50:02.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'87 lcod 11'86 mlcod 11'86 active append_log transaction_applied = 1 2022-03-29T17:50:02.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'87, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.530 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 1 info: 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.439384460s ub 24.494297028s int 16.000000000s))): complete 2022-03-29T17:50:02.530 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,839 [shard 0] osd - replicated_request(id=88, detail=RepRequest(from=1 req=osd_repop(client.4125.0:130 1.0 e12/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 12'88, mlcod=12'88) v3)) start 2022-03-29T17:50:02.530 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,840 [shard 0] osd - replicated_request(id=89, detail=RepRequest(from=1 req=osd_repop(client.4125.0:131 1.0 e12/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 12'89, mlcod=12'89) v3)) start 2022-03-29T17:50:02.530 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,840 [shard 0] osd - replicated_request(id=90, detail=RepRequest(from=1 req=osd_repop(client.4125.0:132 1.0 e12/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 12'90, mlcod=12'90) v3)) start 2022-03-29T17:50:02.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,840 [shard 0] osd - replicated_request(id=91, detail=RepRequest(from=1 req=osd_repop(client.4125.0:133 1.0 e12/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 12'91, mlcod=12'91) v3)) start 2022-03-29T17:50:02.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,841 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'87 (0'0,12'87] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'87 mlcod 12'87 active append_log log((0'0,12'87], crt=12'87) {12'88 (12'87) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:130 2022-03-29T17:50:01.803513+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,841 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'87 lcod 12'87 mlcod 12'87 active add_log_entry 12'88 (12'87) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:130 2022-03-29T17:50:01.803513+0000 0 ObjectCleanRegions clean_offsets: [0~512,516~18446744073709551099], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,841 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'88 lcod 12'87 mlcod 12'87 active append_log approx pg log length = 88 2022-03-29T17:50:02.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,841 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'88 lcod 12'87 mlcod 12'87 active append_log transaction_applied = 1 2022-03-29T17:50:02.532 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,841 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.532 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,841 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'88, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.532 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,841 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,825 [shard 0] osd - client_request(id=135, detail=m=[osd_op(client.4125.0:134 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,841 [shard 0] osd - OpSequencer::start_op: op=135, last_started=134, last_unblocked=134, last_completed=129 2022-03-29T17:50:02.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.477348328s) for pg: 2.4 2022-03-29T17:50:02.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 24.477348328s 2022-03-29T17:50:02.535 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'88 (0'0,12'88] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'88 mlcod 12'88 active append_log log((0'0,12'88], crt=12'88) {12'89 (12'88) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:131 2022-03-29T17:50:01.803533+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.535 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'88 lcod 12'88 mlcod 12'88 active add_log_entry 12'89 (12'88) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:131 2022-03-29T17:50:01.803533+0000 0 ObjectCleanRegions clean_offsets: [0~516,66052~18446744073709485563], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.535 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'89 lcod 12'88 mlcod 12'88 active append_log approx pg log length = 89 2022-03-29T17:50:02.536 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'89 lcod 12'88 mlcod 12'88 active append_log transaction_applied = 1 2022-03-29T17:50:02.536 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.536 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'89, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.477348328s (sent 24.477348328s) 2022-03-29T17:50:02.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.477348328s))): submitting ctx 2022-03-29T17:50:02.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,842 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,843 [shard 0] osd - client_request(id=135, detail=m=[osd_op(client.4125.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,843 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,843 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.540 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,843 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,843 [shard 0] osd - client_request(id=136, detail=m=[osd_op(client.4125.0:135 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,843 [shard 0] osd - client_request(id=136, detail=m=[osd_op(client.4125.0:135 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,843 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:02.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,843 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:02.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'91 lcod 11'86 mlcod 11'86 active+clean calc_trim_to_aggressive limit = 11'86 2022-03-29T17:50:02.543 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'89 (0'0,12'89] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'89 mlcod 12'89 active append_log log((0'0,12'89], crt=12'89) {12'90 (12'89) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:132 2022-03-29T17:50:01.803548+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.544 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'89 lcod 12'89 mlcod 12'89 active add_log_entry 12'90 (12'89) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:132 2022-03-29T17:50:01.803548+0000 0 ObjectCleanRegions clean_offsets: [0~66052,66056~18446744073709485559], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.544 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'90 lcod 12'89 mlcod 12'89 active append_log approx pg log length = 90 2022-03-29T17:50:02.544 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'90 lcod 12'89 mlcod 12'89 active append_log transaction_applied = 1 2022-03-29T17:50:02.544 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.545 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'90, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'91 lcod 11'86 mlcod 11'86 active+clean append_log log((0'0,12'91], crt=12'91) {12'92 (12'91) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:134 2022-03-29T17:50:01.803615+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'91 lcod 11'86 mlcod 11'86 active+clean add_log_entry 12'92 (12'91) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:134 2022-03-29T17:50:01.803615+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'92 lcod 11'86 mlcod 11'86 active+clean append_log approx pg log length = 92 2022-03-29T17:50:02.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'92 lcod 11'86 mlcod 11'86 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,844 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,845 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'92, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,845 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,845 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=0 pgid=2.4 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.477348328s))): complete 2022-03-29T17:50:02.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,845 [shard 0] osd - client_request(id=136, detail=m=[osd_op(client.4125.0:135 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.548 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,845 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,846 [shard 0] osd - OpSequencer::start_op: op=136, last_started=135, last_unblocked=135, last_completed=129 2022-03-29T17:50:02.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,846 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,846 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:02.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,846 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.0 pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): start 2022-03-29T17:50:02.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - replicated_request(id=92, detail=RepRequest(from=1 req=osd_repop(client.4125.0:134 1.0 e12/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 12'92, mlcod=12'92) v3)) start 2022-03-29T17:50:02.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - client_request(id=136, detail=m=[osd_op(client.4125.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.0 pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): got map 12 2022-03-29T17:50:02.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.0 pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): pg present 2022-03-29T17:50:02.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:02.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.554 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'90 (0'0,12'90] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'90 mlcod 12'90 active append_log log((0'0,12'90], crt=12'90) {12'91 (12'90) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:133 2022-03-29T17:50:01.803591+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.554 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'90 lcod 12'90 mlcod 12'90 active add_log_entry 12'91 (12'90) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:133 2022-03-29T17:50:01.803591+0000 0 ObjectCleanRegions clean_offsets: [0~66056,66060~18446744073709485555], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.555 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'91 lcod 12'90 mlcod 12'90 active append_log approx pg log length = 91 2022-03-29T17:50:02.555 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'91 lcod 12'90 mlcod 12'90 active append_log transaction_applied = 1 2022-03-29T17:50:02.555 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.555 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,847 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'91, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.556 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:02.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'92 lcod 11'86 mlcod 11'86 active+clean calc_trim_to_aggressive limit = 11'86 2022-03-29T17:50:02.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'92 lcod 11'86 mlcod 11'86 active+clean append_log log((0'0,12'92], crt=12'92) {12'93 (12'92) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:135 2022-03-29T17:50:01.803629+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'92 lcod 11'86 mlcod 11'86 active+clean add_log_entry 12'93 (12'92) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:135 2022-03-29T17:50:01.803629+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'93 lcod 11'86 mlcod 11'86 active+clean append_log approx pg log length = 93 2022-03-29T17:50:02.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=11'86 lua=0'0 crt=12'93 lcod 11'86 mlcod 11'86 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,848 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'93, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,849 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.561 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'91 (0'0,12'91] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'91 mlcod 12'91 active append_log log((0'0,12'91], crt=12'91) {12'92 (12'91) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:134 2022-03-29T17:50:01.803615+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.561 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'91 lcod 12'91 mlcod 12'91 active add_log_entry 12'92 (12'91) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:134 2022-03-29T17:50:01.803615+0000 0 ObjectCleanRegions clean_offsets: [0~66060,131596~18446744073709420019], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.561 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'92 lcod 12'91 mlcod 12'91 active append_log approx pg log length = 92 2022-03-29T17:50:02.561 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'92 lcod 12'91 mlcod 12'91 active append_log transaction_applied = 1 2022-03-29T17:50:02.562 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.562 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'92, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.562 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.562 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - replicated_request(id=93, detail=RepRequest(from=1 req=osd_repop(client.4125.0:135 1.0 e12/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 12'93, mlcod=12'93) v3)) start 2022-03-29T17:50:02.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,849 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.0 pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:02.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s) 2022-03-29T17:50:02.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 24.546831131s 2022-03-29T17:50:02.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,850 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 24.546831131s 2022-03-29T17:50:02.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,851 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,851 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.0 pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,851 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:02.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,851 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): start 2022-03-29T17:50:02.566 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,851 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:02.567 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,851 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.436038971s))): start 2022-03-29T17:50:02.567 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,852 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.436038971s))): got map 12 2022-03-29T17:50:02.567 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,852 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.436038971s))): pg present 2022-03-29T17:50:02.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,852 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.0 pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): complete 2022-03-29T17:50:02.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,852 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): got map 12 2022-03-29T17:50:02.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,852 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): pg present 2022-03-29T17:50:02.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,852 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:02.570 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'92 (0'0,12'92] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'92 mlcod 12'92 active append_log log((0'0,12'92], crt=12'92) {12'93 (12'92) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:135 2022-03-29T17:50:01.803629+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.570 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'92 lcod 12'92 mlcod 12'92 active add_log_entry 12'93 (12'92) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:135 2022-03-29T17:50:01.803629+0000 0 ObjectCleanRegions clean_offsets: [0~131596,131600~18446744073709420015], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.570 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'93 lcod 12'92 mlcod 12'92 active append_log approx pg log length = 93 2022-03-29T17:50:02.571 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'93 lcod 12'92 mlcod 12'92 active append_log transaction_applied = 1 2022-03-29T17:50:02.571 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,852 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11 2022-03-29T17:50:02.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:02.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): start 2022-03-29T17:50:02.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,853 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'93, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.436038971s) for pg: 2.2 2022-03-29T17:50:02.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 24.436038971s 2022-03-29T17:50:02.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.436038971s (sent 24.436038971s) 2022-03-29T17:50:02.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): got map 12 2022-03-29T17:50:02.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): complete 2022-03-29T17:50:02.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): pg present 2022-03-29T17:50:02.577 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.436038971s))): submitting ctx 2022-03-29T17:50:02.577 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,854 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,855 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:02.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,855 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11 2022-03-29T17:50:02.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,855 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:50:02.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,855 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,855 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,855 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,855 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,855 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.2 pg_lease_ack(ruub 24.472705841s))): start 2022-03-29T17:50:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,856 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,856 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.485576630s))): start 2022-03-29T17:50:02.582 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,856 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=1 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.436038971s))): complete 2022-03-29T17:50:02.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,856 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.2 pg_lease_ack(ruub 24.472705841s))): got map 12 2022-03-29T17:50:02.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.2 pg_lease_ack(ruub 24.472705841s))): pg present 2022-03-29T17:50:02.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=0 pgid=2.2 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 0 info: 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.330034256s ub 24.436038971s int 16.000000000s))): complete 2022-03-29T17:50:02.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.485576630s))): got map 12 2022-03-29T17:50:02.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.2 pg_lease_ack(ruub 24.472705841s) for pg: 2.3 2022-03-29T17:50:02.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 24.472705841s 2022-03-29T17:50:02.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.472705841s (sent 24.472705841s) 2022-03-29T17:50:02.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.2 pg_lease_ack(ruub 24.472705841s))): submitting ctx 2022-03-29T17:50:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,857 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.485576630s))): pg present 2022-03-29T17:50:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.485576630s) for pg: 2.5 2022-03-29T17:50:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 24.485576630s 2022-03-29T17:50:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.485576630s (sent 24.485576630s) 2022-03-29T17:50:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.485576630s))): submitting ctx 2022-03-29T17:50:02.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:02.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,858 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): start 2022-03-29T17:50:02.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): got map 12 2022-03-29T17:50:02.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): pg present 2022-03-29T17:50:02.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.3 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.2 pg_lease_ack(ruub 24.472705841s))): complete 2022-03-29T17:50:02.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:02.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s) 2022-03-29T17:50:02.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 24.555475235s 2022-03-29T17:50:02.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 24.555475235s 2022-03-29T17:50:02.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:02.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,859 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.490119934s))): start 2022-03-29T17:50:02.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,860 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:02.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,860 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): start 2022-03-29T17:50:02.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,860 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.490119934s))): got map 12 2022-03-29T17:50:02.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,860 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.490119934s))): pg present 2022-03-29T17:50:02.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,860 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): got map 12 2022-03-29T17:50:02.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): pg present 2022-03-29T17:50:02.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:02.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11 2022-03-29T17:50:02.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.490119934s) for pg: 2.7 2022-03-29T17:50:02.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 24.490119934s 2022-03-29T17:50:02.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.490119934s (sent 24.490119934s) 2022-03-29T17:50:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.490119934s))): submitting ctx 2022-03-29T17:50:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,861 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=0 pgid=2.5 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.485576630s))): complete 2022-03-29T17:50:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,862 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:02.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,862 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): start 2022-03-29T17:50:02.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,862 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): got map 12 2022-03-29T17:50:02.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,862 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:02.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,862 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.494297028s))): start 2022-03-29T17:50:02.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,862 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): pg present 2022-03-29T17:50:02.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,862 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:02.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): start 2022-03-29T17:50:02.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.494297028s))): got map 12 2022-03-29T17:50:02.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.494297028s))): pg present 2022-03-29T17:50:02.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): got map 12 2022-03-29T17:50:02.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:02.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11 2022-03-29T17:50:02.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:50:02.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.612 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,863 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): pg present 2022-03-29T17:50:02.612 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.494297028s) for pg: 2.6 2022-03-29T17:50:02.612 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 24.494297028s 2022-03-29T17:50:02.612 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.494297028s (sent 24.494297028s) 2022-03-29T17:50:02.612 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.494297028s))): submitting ctx 2022-03-29T17:50:02.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:02.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s) 2022-03-29T17:50:02.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 24.560108185s 2022-03-29T17:50:02.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=0/0 les/c/f=0/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 24.560108185s 2022-03-29T17:50:02.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:02.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): start 2022-03-29T17:50:02.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): complete 2022-03-29T17:50:02.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): got map 12 2022-03-29T17:50:02.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,864 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): pg present 2022-03-29T17:50:02.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:02.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): start 2022-03-29T17:50:02.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): got map 12 2022-03-29T17:50:02.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:02.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11 2022-03-29T17:50:02.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): pg present 2022-03-29T17:50:02.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:02.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history advanced history from ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11 2022-03-29T17:50:02.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:50:02.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): submitting ctx 2022-03-29T17:50:02.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,865 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,866 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:129 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,866 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,866 [shard 0] osd - OpSequencer::finish_op_in_order: op=130, last_started=136, last_unblocked=136, last_completed=129 2022-03-29T17:50:02.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,866 [shard 0] osd - client_request(id=130, detail=m=[osd_op(client.4125.0:129 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~512 in=512b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.620 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,867 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:130 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,867 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,867 [shard 0] osd - OpSequencer::finish_op_in_order: op=131, last_started=136, last_unblocked=136, last_completed=130 2022-03-29T17:50:02.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,867 [shard 0] osd - client_request(id=131, detail=m=[osd_op(client.4125.0:130 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 512~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,867 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:131 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.622 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,867 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.622 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,867 [shard 0] osd - OpSequencer::finish_op_in_order: op=132, last_started=136, last_unblocked=136, last_completed=131 2022-03-29T17:50:02.622 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,867 [shard 0] osd - client_request(id=132, detail=m=[osd_op(client.4125.0:131 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 516~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.622 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,868 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,868 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,868 [shard 0] osd - OpSequencer::finish_op_in_order: op=133, last_started=136, last_unblocked=136, last_completed=132 2022-03-29T17:50:02.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,868 [shard 0] osd - client_request(id=133, detail=m=[osd_op(client.4125.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66052~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,868 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,868 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,868 [shard 0] osd - OpSequencer::finish_op_in_order: op=134, last_started=136, last_unblocked=136, last_completed=133 2022-03-29T17:50:02.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,869 [shard 0] osd - client_request(id=134, detail=m=[osd_op(client.4125.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66056~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,869 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,869 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,869 [shard 0] osd - OpSequencer::finish_op_in_order: op=135, last_started=136, last_unblocked=136, last_completed=134 2022-03-29T17:50:02.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,869 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=0 pgid=2.7 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.490119934s))): complete 2022-03-29T17:50:02.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,869 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): complete 2022-03-29T17:50:02.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,869 [shard 0] osd - client_request(id=135, detail=m=[osd_op(client.4125.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 66060~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,869 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.353706360s ub 24.449274063s int 16.000000000s))): complete 2022-03-29T17:50:02.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,870 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,870 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=0 pgid=2.6 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.0 pg_lease_ack(ruub 24.494297028s))): complete 2022-03-29T17:50:02.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,870 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLease epoch 12 from osd.2 pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): complete 2022-03-29T17:50:02.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,870 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,870 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/0 les/c/f=12/0/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): complete 2022-03-29T17:50:02.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,870 [shard 0] osd - OpSequencer::finish_op_in_order: op=136, last_started=136, last_unblocked=136, last_completed=135 2022-03-29T17:50:02.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,870 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MInfoRec from 2 info: 2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) pg_lease(ru 24.358369827s ub 24.453264236s int 16.000000000s))): complete 2022-03-29T17:50:02.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,870 [shard 0] osd - client_request(id=136, detail=m=[osd_op(client.4125.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 131596~4 in=4b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,896 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:02.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,896 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.449274063s))): start 2022-03-29T17:50:02.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,896 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.449274063s))): got map 12 2022-03-29T17:50:02.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,896 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.449274063s))): pg present 2022-03-29T17:50:02.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.449274063s) for pg: 2.0 2022-03-29T17:50:02.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 24.449274063s 2022-03-29T17:50:02.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.449274063s (sent 24.449274063s) 2022-03-29T17:50:02.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.449274063s))): submitting ctx 2022-03-29T17:50:02.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:02.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.453264236s))): start 2022-03-29T17:50:02.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.453264236s))): got map 12 2022-03-29T17:50:02.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - peering_event(id=20, detail=PeeringEvent(from=1 pgid=2.0 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.449274063s))): complete 2022-03-29T17:50:02.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,897 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.453264236s))): pg present 2022-03-29T17:50:02.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,898 [shard 0] osd - do_peering_event handling epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.453264236s) for pg: 2.1 2022-03-29T17:50:02.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,898 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 24.453264236s 2022-03-29T17:50:02.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,898 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.453264236s (sent 24.453264236s) 2022-03-29T17:50:02.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,898 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,898 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.453264236s))): submitting ctx 2022-03-29T17:50:02.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,898 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:01,898 [shard 0] osd - peering_event(id=21, detail=PeeringEvent(from=1 pgid=2.1 sent=12 requested=12 evt=epoch_sent: 12 epoch_requested: 12 MLeaseAck epoch 12 from osd.1 pg_lease_ack(ruub 24.453264236s))): complete 2022-03-29T17:50:02.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,907 [shard 0] osd - client_request(id=137, detail=m=[osd_op(client.4125.0:136 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,907 [shard 0] osd - client_request(id=137, detail=m=[osd_op(client.4125.0:136 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - client_request(id=137, detail=m=[osd_op(client.4125.0:136 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - OpSequencer::start_op: op=137, last_started=136, last_unblocked=136, last_completed=136 2022-03-29T17:50:02.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - client_request(id=137, detail=m=[osd_op(client.4125.0:136 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:02.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - client_request(id=138, detail=m=[osd_op(client.4125.0:137 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - client_request(id=138, detail=m=[osd_op(client.4125.0:137 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:02.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:02.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:02.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'93 mlcod 12'93 active+clean calc_trim_to_aggressive limit = 12'93 2022-03-29T17:50:02.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'93 mlcod 12'93 active+clean append_log log((0'0,12'93], crt=12'93) {12'94 (11'86) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:136 2022-03-29T17:50:01.907523+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'93 lua=0'0 crt=12'93 lcod 12'93 mlcod 12'93 active+clean add_log_entry 12'94 (11'86) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:136 2022-03-29T17:50:01.907523+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'93 lua=0'0 crt=12'94 lcod 12'93 mlcod 12'93 active+clean append_log approx pg log length = 94 2022-03-29T17:50:02.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'93 lua=0'0 crt=12'94 lcod 12'93 mlcod 12'93 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.643 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,908 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.643 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,909 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'94, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.643 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,909 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.644 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,910 [shard 0] osd - replicated_request(id=94, detail=RepRequest(from=1 req=osd_repop(client.4125.0:136 1.0 e12/8 1:30306672:devicehealth::main.db.0000000000000000:head v 12'94, mlcod=12'94) v3)) start 2022-03-29T17:50:02.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,910 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'93 (0'0,12'93] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'93 mlcod 12'93 active append_log log((0'0,12'93], crt=12'93) {12'94 (11'86) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:136 2022-03-29T17:50:01.907523+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,910 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'93 lcod 12'93 mlcod 12'93 active add_log_entry 12'94 (11'86) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:136 2022-03-29T17:50:01.907523+0000 0 ObjectCleanRegions clean_offsets: [65536~18446744073709486079], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,910 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'94 lcod 12'93 mlcod 12'93 active append_log approx pg log length = 94 2022-03-29T17:50:02.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,910 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'94 lcod 12'93 mlcod 12'93 active append_log transaction_applied = 1 2022-03-29T17:50:02.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,910 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,910 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'94, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,910 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,911 [shard 0] osd - client_request(id=138, detail=m=[osd_op(client.4125.0:137 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,911 [shard 0] osd - OpSequencer::start_op: op=138, last_started=137, last_unblocked=137, last_completed=136 2022-03-29T17:50:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,911 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - client_request(id=138, detail=m=[osd_op(client.4125.0:137 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:02.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op write 2022-03-29T17:50:02.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - handling op write on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'93 lua=0'0 crt=12'94 lcod 12'93 mlcod 12'93 active+clean calc_trim_to_aggressive limit = 12'93 2022-03-29T17:50:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'93 lua=0'0 crt=12'94 lcod 12'93 mlcod 12'93 active+clean append_log log((0'0,12'94], crt=12'94) {12'95 (12'94) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:137 2022-03-29T17:50:01.907574+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'93 lua=0'0 crt=12'94 lcod 12'93 mlcod 12'93 active+clean add_log_entry 12'95 (12'94) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:137 2022-03-29T17:50:01.907574+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'93 lua=0'0 crt=12'95 lcod 12'93 mlcod 12'93 active+clean append_log approx pg log length = 95 2022-03-29T17:50:02.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'93 lua=0'0 crt=12'95 lcod 12'93 mlcod 12'93 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'95, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,912 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.654 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,913 [shard 0] osd - replicated_request(id=95, detail=RepRequest(from=1 req=osd_repop(client.4125.0:137 1.0 e12/8 1:30306672:devicehealth::main.db.0000000000000000:head v 12'95, mlcod=12'95) v3)) start 2022-03-29T17:50:02.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,914 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'94 (0'0,12'94] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'94 mlcod 12'94 active append_log log((0'0,12'94], crt=12'94) {12'95 (12'94) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:137 2022-03-29T17:50:01.907574+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,914 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'94 lcod 12'94 mlcod 12'94 active add_log_entry 12'95 (12'94) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:137 2022-03-29T17:50:01.907574+0000 0 ObjectCleanRegions clean_offsets: [0~65536,131072~18446744073709420543], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,914 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'95 lcod 12'94 mlcod 12'94 active append_log approx pg log length = 95 2022-03-29T17:50:02.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,914 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'95 lcod 12'94 mlcod 12'94 active append_log transaction_applied = 1 2022-03-29T17:50:02.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,914 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.656 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,914 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'95, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.656 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,914 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,915 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:136 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,915 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,915 [shard 0] osd - OpSequencer::finish_op_in_order: op=137, last_started=138, last_unblocked=138, last_completed=136 2022-03-29T17:50:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,915 [shard 0] osd - client_request(id=137, detail=m=[osd_op(client.4125.0:136 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 0~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,916 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:137 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,916 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,916 [shard 0] osd - OpSequencer::finish_op_in_order: op=138, last_started=138, last_unblocked=138, last_completed=137 2022-03-29T17:50:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,916 [shard 0] osd - client_request(id=138, detail=m=[osd_op(client.4125.0:137 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {write 65536~65536 in=65536b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,956 [shard 0] osd - client_request(id=139, detail=m=[osd_op(client.4125.0:138 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,956 [shard 0] osd - client_request(id=139, detail=m=[osd_op(client.4125.0:138 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,956 [shard 0] osd - client_request(id=139, detail=m=[osd_op(client.4125.0:138 1.0 1.756970d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,956 [shard 0] osd - OpSequencer::start_op: op=139, last_started=138, last_unblocked=138, last_completed=138 2022-03-29T17:50:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - do_recover_missing check for recovery, 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - client_request(id=139, detail=m=[osd_op(client.4125.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - with_head_obc 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - with_head_obc: found 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head in cache 2022-03-29T17:50:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head - handling op write 2022-03-29T17:50:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - handling op write on object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head all operations successful 2022-03-29T17:50:02.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - do_osd_ops_execute: object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head submitting txn 2022-03-29T17:50:02.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'95 mlcod 12'95 active+clean calc_trim_to_aggressive limit = 12'95 2022-03-29T17:50:02.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'95 mlcod 12'95 active+clean append_log log((0'0,12'95], crt=12'95) {12'96 (12'93) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:138 2022-03-29T17:50:01.956447+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'95 lua=0'0 crt=12'95 lcod 12'95 mlcod 12'95 active+clean add_log_entry 12'96 (12'93) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:138 2022-03-29T17:50:01.956447+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'95 lua=0'0 crt=12'96 lcod 12'95 mlcod 12'95 active+clean append_log approx pg log length = 96 2022-03-29T17:50:02.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'95 lua=0'0 crt=12'96 lcod 12'95 mlcod 12'95 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'96, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,957 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,958 [shard 0] osd - replicated_request(id=96, detail=RepRequest(from=1 req=osd_repop(client.4125.0:138 1.0 e12/8 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 12'96, mlcod=12'96) v3)) start 2022-03-29T17:50:02.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,958 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'95 (0'0,12'95] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'95 mlcod 12'95 active append_log log((0'0,12'95], crt=12'95) {12'96 (12'93) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:138 2022-03-29T17:50:01.956447+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,958 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'95 lcod 12'95 mlcod 12'95 active add_log_entry 12'96 (12'93) modify 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head by client.4125.0:138 2022-03-29T17:50:01.956447+0000 0 ObjectCleanRegions clean_offsets: [28~18446744073709551587], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,958 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'96 lcod 12'95 mlcod 12'95 active append_log approx pg log length = 96 2022-03-29T17:50:02.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,958 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'96 lcod 12'95 mlcod 12'95 active append_log transaction_applied = 1 2022-03-29T17:50:02.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,959 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,959 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'96, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:01,959 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,999 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head sending reply 2022-03-29T17:50:02.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:01,999 [shard 0] osd - with_head_obc: released 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head 2022-03-29T17:50:02.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - OpSequencer::finish_op_in_order: op=139, last_started=139, last_unblocked=139, last_completed=138 2022-03-29T17:50:02.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - client_request(id=139, detail=m=[osd_op(client.4125.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head {write 0~28 in=28b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - client_request(id=140, detail=m=[osd_op(client.4125.0:139 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): start 2022-03-29T17:50:02.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - client_request(id=140, detail=m=[osd_op(client.4125.0:139 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): in repeat 2022-03-29T17:50:02.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - client_request(id=140, detail=m=[osd_op(client.4125.0:139 1.0 1.4e660c0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e9) v8]) same_interval_since: 8 2022-03-29T17:50:02.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - OpSequencer::start_op: op=140, last_started=139, last_unblocked=139, last_completed=139 2022-03-29T17:50:02.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - do_recover_missing check for recovery, 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - client_request(id=140, detail=m=[osd_op(client.4125.0:139 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): got obc lock 2022-03-29T17:50:02.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - with_head_obc 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - with_head_obc: found 1:30306672:devicehealth::main.db.0000000000000000:head in cache 2022-03-29T17:50:02.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,000 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op cmpxattr 2022-03-29T17:50:02.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - handling op cmpxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - cmpxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:02.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - cmpxattr lhs=172.21.15.138:0/3250333293, rhs=172.21.15.138:0/3250333293 2022-03-29T17:50:02.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op setxattr 2022-03-29T17:50:02.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.678 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:02,001 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:02.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_striper.excl 2022-03-29T17:50:02.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head - handling op call 2022-03-29T17:50:02.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - handling op call on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - calling method lock.unlock, num_read=1, num_write=1 2022-03-29T17:50:02.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - handling op getxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - getxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:02.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - handling op setxattr on object 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:02,001 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:50:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - setxattr on obj=1:30306672:devicehealth::main.db.0000000000000000:head for attr=_lock.striper.lock 2022-03-29T17:50:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,001 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:50:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head all operations successful 2022-03-29T17:50:02.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - do_osd_ops_execute: object 1:30306672:devicehealth::main.db.0000000000000000:head submitting txn 2022-03-29T17:50:02.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'96 mlcod 12'96 active+clean calc_trim_to_aggressive limit = 12'96 2022-03-29T17:50:02.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'96 mlcod 12'96 active+clean append_log log((0'0,12'96], crt=12'96) {12'97 (12'95) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:139 2022-03-29T17:50:01.999943+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'96 lua=0'0 crt=12'96 lcod 12'96 mlcod 12'96 active+clean add_log_entry 12'97 (12'95) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:139 2022-03-29T17:50:01.999943+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'96 lua=0'0 crt=12'97 lcod 12'96 mlcod 12'96 active+clean append_log approx pg log length = 97 2022-03-29T17:50:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 luod=12'96 lua=0'0 crt=12'97 lcod 12'96 mlcod 12'96 active+clean append_log transaction_applied = 1 2022-03-29T17:50:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'97, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,002 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:50:02.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:02,040 [shard 0] osd - replicated_request(id=97, detail=RepRequest(from=1 req=osd_repop(client.4125.0:139 1.0 e12/8 1:30306672:devicehealth::main.db.0000000000000000:head v 12'97, mlcod=12'97) v3)) start 2022-03-29T17:50:02.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:02,040 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'96 (0'0,12'96] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'96 mlcod 12'96 active append_log log((0'0,12'96], crt=12'96) {12'97 (12'95) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:139 2022-03-29T17:50:01.999943+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:50:02.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:02,040 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'96 lcod 12'96 mlcod 12'96 active add_log_entry 12'97 (12'95) modify 1:30306672:devicehealth::main.db.0000000000000000:head by client.4125.0:139 2022-03-29T17:50:01.999943+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:50:02.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:02,041 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 lcod 12'96 mlcod 12'96 active append_log approx pg log length = 97 2022-03-29T17:50:02.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:02,041 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 lcod 12'96 mlcod 12'96 active append_log transaction_applied = 1 2022-03-29T17:50:02.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:02,041 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:50:02.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:02,041 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 12'97, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:50:02.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:02,041 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:50:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,081 [shard 0] osd - do_osd_ops: osd_op(client.4125.0:139 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8 - object 1:30306672:devicehealth::main.db.0000000000000000:head sending reply 2022-03-29T17:50:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,082 [shard 0] osd - with_head_obc: released 1:30306672:devicehealth::main.db.0000000000000000:head 2022-03-29T17:50:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,082 [shard 0] osd - OpSequencer::finish_op_in_order: op=140, last_started=140, last_unblocked=140, last_completed=139 2022-03-29T17:50:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,082 [shard 0] osd - client_request(id=140, detail=m=[osd_op(client.4125.0:139 1.0 1:30306672:devicehealth::main.db.0000000000000000:head {cmpxattr striper.excl (26) op 1 mode 1 in=38b, setxattr striper.excl in=12b, call lock.unlock in=72b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e9) v8]): destroying 2022-03-29T17:50:02.688 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:02,229 [shard 0] alienstore - stat 2022-03-29T17:50:02.688 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:02,234 [shard 0] alienstore - stat 2022-03-29T17:50:02.689 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:02,237 [shard 0] alienstore - stat 2022-03-29T17:50:02.718 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:50:02.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:02,716 [shard 0] osd - handle_osd_map osd_map(12..13 src has 1..13) v4 2022-03-29T17:50:02.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:02,716 [shard 0] osd - handle_osd_map epochs [12..13], i have 12, src has [1..13] 2022-03-29T17:50:02.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,717 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:02.731 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:02,719 [shard 0] osd - osd.1: committed_osd_maps(13, 13) 2022-03-29T17:50:02.731 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:02,719 [shard 0] osd - osd.1: map e 13 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:02.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=19, detail=PGAdvanceMap(pg=1.0 from=12 to=13)): start 2022-03-29T17:50:02.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=20, detail=PGAdvanceMap(pg=2.0 from=12 to=13)): start 2022-03-29T17:50:02.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=21, detail=PGAdvanceMap(pg=2.1 from=12 to=13)): start 2022-03-29T17:50:02.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=22, detail=PGAdvanceMap(pg=2.2 from=12 to=13)): start 2022-03-29T17:50:02.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=23, detail=PGAdvanceMap(pg=2.3 from=12 to=13)): start 2022-03-29T17:50:02.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=24, detail=PGAdvanceMap(pg=2.4 from=12 to=13)): start 2022-03-29T17:50:02.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=25, detail=PGAdvanceMap(pg=2.5 from=12 to=13)): start 2022-03-29T17:50:02.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=26, detail=PGAdvanceMap(pg=2.6 from=12 to=13)): start 2022-03-29T17:50:02.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_advance_map(id=27, detail=PGAdvanceMap(pg=2.7 from=12 to=13)): start 2022-03-29T17:50:02.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:50:02.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:50:02.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:50:02.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:50:02.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:50:02.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:50:02.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:02.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:50:02.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,719 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:02.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:50:02.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:50:02.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:02.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:50:02.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:50:02.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:50:02.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:9 2022-03-29T17:50:02.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:02.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:50:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:50:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:50:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:50:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:50:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:9 2022-03-29T17:50:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:50:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:50:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:50:02.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:9 2022-03-29T17:50:02.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,720 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:50:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:50:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:50:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:02.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active advmap 2022-03-29T17:50:02.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Started advmap 2022-03-29T17:50:02.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:02.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:50:02.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:50:02.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:50:02.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:50:02.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:02.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:50:02.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:50:02.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:02.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map 2022-03-29T17:50:02.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active: handling ActMap 2022-03-29T17:50:02.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: handle ActMap primary 2022-03-29T17:50:02.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean publish_stats_to_osd 12:12 2022-03-29T17:50:02.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:50:02.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:50:02.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:50:02.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.751 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:9 2022-03-29T17:50:02.751 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,721 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.751 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.751 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:02.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:50:02.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:50:02.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:02.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:50:02.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:50:02.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:50:02.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:02.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:9 2022-03-29T17:50:02.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:50:02.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:02.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - log is not dirty 2022-03-29T17:50:02.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:02.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_advance_map(id=22, detail=PGAdvanceMap(pg=2.2 from=12 to=13)): complete 2022-03-29T17:50:02.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_advance_map(id=26, detail=PGAdvanceMap(pg=2.6 from=12 to=13)): complete 2022-03-29T17:50:02.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_advance_map(id=23, detail=PGAdvanceMap(pg=2.3 from=12 to=13)): complete 2022-03-29T17:50:02.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_advance_map(id=27, detail=PGAdvanceMap(pg=2.7 from=12 to=13)): complete 2022-03-29T17:50:02.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_advance_map(id=20, detail=PGAdvanceMap(pg=2.0 from=12 to=13)): complete 2022-03-29T17:50:02.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,722 [shard 0] osd - pg_advance_map(id=19, detail=PGAdvanceMap(pg=1.0 from=12 to=13)): complete 2022-03-29T17:50:02.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,723 [shard 0] osd - pg_advance_map(id=25, detail=PGAdvanceMap(pg=2.5 from=12 to=13)): complete 2022-03-29T17:50:02.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,723 [shard 0] osd - pg_advance_map(id=21, detail=PGAdvanceMap(pg=2.1 from=12 to=13)): complete 2022-03-29T17:50:02.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:02,723 [shard 0] osd - pg_advance_map(id=24, detail=PGAdvanceMap(pg=2.4 from=12 to=13)): complete 2022-03-29T17:50:02.757 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:02,723 [shard 0] osd - osd.1: now active 2022-03-29T17:50:02.758 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":35195},{"type":"v1","addr":"172.21.15.138:6813","nonce":35195}]},"active_addr":"172.21.15.138:6813/35195","active_change":"2022-03-29T17:49:56.681076+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":3250333293}]},{"addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":3450850499}]},{"addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":3541739701}]}]}} 2022-03-29T17:50:02.761 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-03-29T17:50:02.761 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-03-29T17:50:02.762 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-29T17:50:03.105 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:50:03.105 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":13,"fsid":"3be68405-f255-4011-9a3b-ba646329d493","created":"2022-03-29T17:49:51.193768+0000","modified":"2022-03-29T17:50:02.711306+0000","last_up_change":"2022-03-29T17:49:55.205833+0000","last_in_change":"2022-03-29T17:49:52.178337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-29T17:49:56.727900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-03-29T17:50:00.248337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3e4be414-1d2c-4bc5-b6b2-d1f7948d012a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":35577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":35577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":35577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":35577}]},"public_addr":"172.21.15.138:6805/35577","cluster_addr":"172.21.15.138:6804/35577","heartbeat_back_addr":"172.21.15.138:6806/35577","heartbeat_front_addr":"172.21.15.138:6809/35577","state":["exists","up"]},{"osd":1,"uuid":"59cc9d2e-e40e-4048-b5cb-31db1b464c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":35578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":35578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":35578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":35578}]},"public_addr":"172.21.15.138:6800/35578","cluster_addr":"172.21.15.138:6801/35578","heartbeat_back_addr":"172.21.15.138:6807/35578","heartbeat_front_addr":"172.21.15.138:6808/35578","state":["exists","up"]},{"osd":2,"uuid":"88a5bd02-6c7b-4069-9fdb-aea1c201b934","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":35579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":35579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":35579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":35579}]},"public_addr":"172.21.15.138:6803/35579","cluster_addr":"172.21.15.138:6802/35579","heartbeat_back_addr":"172.21.15.138:6811/35579","heartbeat_front_addr":"172.21.15.138:6810/35579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-29T17:50:03.119 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-03-29T17:50:03.119 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-29T17:50:03.233 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,231 [shard 0] alienstore - stat 2022-03-29T17:50:03.241 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,240 [shard 0] alienstore - stat 2022-03-29T17:50:03.242 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,240 [shard 0] alienstore - stat 2022-03-29T17:50:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:50:03.427 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":13,"fsid":"3be68405-f255-4011-9a3b-ba646329d493","created":"2022-03-29T17:49:51.193768+0000","modified":"2022-03-29T17:50:02.711306+0000","last_up_change":"2022-03-29T17:49:55.205833+0000","last_in_change":"2022-03-29T17:49:52.178337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-29T17:49:56.727900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-03-29T17:50:00.248337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3e4be414-1d2c-4bc5-b6b2-d1f7948d012a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":35577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":35577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":35577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":35577}]},"public_addr":"172.21.15.138:6805/35577","cluster_addr":"172.21.15.138:6804/35577","heartbeat_back_addr":"172.21.15.138:6806/35577","heartbeat_front_addr":"172.21.15.138:6809/35577","state":["exists","up"]},{"osd":1,"uuid":"59cc9d2e-e40e-4048-b5cb-31db1b464c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":35578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":35578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":35578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":35578}]},"public_addr":"172.21.15.138:6800/35578","cluster_addr":"172.21.15.138:6801/35578","heartbeat_back_addr":"172.21.15.138:6807/35578","heartbeat_front_addr":"172.21.15.138:6808/35578","state":["exists","up"]},{"osd":2,"uuid":"88a5bd02-6c7b-4069-9fdb-aea1c201b934","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":35579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":35579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":35579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":35579}]},"public_addr":"172.21.15.138:6803/35579","cluster_addr":"172.21.15.138:6802/35579","heartbeat_back_addr":"172.21.15.138:6811/35579","heartbeat_front_addr":"172.21.15.138:6810/35579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-29T17:50:03.441 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-03-29T17:50:03.442 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-03-29T17:50:03.443 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-03-29T17:50:03.586 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,584 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> unknown.? -@48832] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:48832/0 2022-03-29T17:50:03.587 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,584 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> unknown.? -@41128] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:41128/0 2022-03-29T17:50:03.587 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,585 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.? -@41128] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:50:03.587 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,586 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.? -@48832] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:50:03.588 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,586 [shard 0] monc - added challenge on [osd.2(client) v2:172.21.15.138:6803/35579 >> client.? -@41128] 2022-03-29T17:50:03.589 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,586 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> unknown.? -@36826] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:36826/0 2022-03-29T17:50:03.589 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,587 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> client.? -@36826] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:50:03.589 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,587 [shard 0] monc - added challenge on [osd.0(client) v2:172.21.15.138:6805/35577 >> client.? -@36826] 2022-03-29T17:50:03.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:03,587 [shard 0] osd - handle_authentication client.admin has caps allow * 2022-03-29T17:50:03.590 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,588 [shard 0] monc - added challenge on [osd.1(client) v2:172.21.15.138:6800/35578 >> client.? -@48832] 2022-03-29T17:50:03.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:03,588 [shard 0] osd - handle_authentication client.admin has caps allow * 2022-03-29T17:50:03.591 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,589 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> client.4158 172.21.15.138:0/3941588267@36826] established: gs=0, pgs=3, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:03.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:03,589 [shard 0] osd - handle_authentication client.admin has caps allow * 2022-03-29T17:50:03.592 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,590 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4157 172.21.15.138:0/768059634@41128] established: gs=0, pgs=3, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:03.592 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,591 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.4156 172.21.15.138:0/727207513@48832] established: gs=0, pgs=3, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:03.598 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,597 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> client.4158 172.21.15.138:0/3941588267@36826] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:50:03.599 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,597 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> client.4158 172.21.15.138:0/3941588267@36826] closing: reset yes, replace no 2022-03-29T17:50:03.599 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:50:03,597 [shard 0] osd - ms_handle_reset 2022-03-29T17:50:03.600 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,597 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.4156 172.21.15.138:0/727207513@48832] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:50:03.600 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,597 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.4156 172.21.15.138:0/727207513@48832] closing: reset yes, replace no 2022-03-29T17:50:03.600 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:50:03,597 [shard 0] osd - ms_handle_reset 2022-03-29T17:50:03.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,598 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4157 172.21.15.138:0/768059634@41128] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:50:03.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,598 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4157 172.21.15.138:0/768059634@41128] closing: reset yes, replace no 2022-03-29T17:50:03.601 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:50:03,598 [shard 0] osd - ms_handle_reset 2022-03-29T17:50:03.612 INFO:teuthology.orchestra.run.smithi138.stdout:30064771080 2022-03-29T17:50:03.613 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-03-29T17:50:03.614 INFO:teuthology.orchestra.run.smithi138.stdout:30064771080 2022-03-29T17:50:03.615 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-03-29T17:50:03.616 INFO:teuthology.orchestra.run.smithi138.stdout:30064771080 2022-03-29T17:50:03.616 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-03-29T17:50:03.730 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,728 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@52685 >> mgr.4102 v2:172.21.15.138:6812/35195] closing: reset no, replace no 2022-03-29T17:50:03.731 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,729 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@51139 >> mgr.4102 v2:172.21.15.138:6812/35195] closing: reset no, replace no 2022-03-29T17:50:03.731 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,729 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@54730 >> mgr.4102 v2:172.21.15.138:6812/35195] closing: reset no, replace no 2022-03-29T17:50:03.732 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:03,730 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@52685 >> mgr.4102 v2:172.21.15.138:6812/35195] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:50:03.732 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:03,730 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@51139 >> mgr.4102 v2:172.21.15.138:6812/35195] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:50:03.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:03,731 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@54730 >> mgr.4102 v2:172.21.15.138:6812/35195] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:50:03.778 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:50:03,776 [shard 0] mgrc - report: no conn available; raport skipped 2022-03-29T17:50:03.779 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:50:03,776 [shard 0] mgrc - report: no conn available; raport skipped 2022-03-29T17:50:03.780 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:50:03,777 [shard 0] mgrc - report: no conn available; raport skipped 2022-03-29T17:50:03.984 INFO:teuthology.orchestra.run.smithi138.stdout:30064771080 2022-03-29T17:50:03.995 INFO:teuthology.orchestra.run.smithi138.stdout:30064771080 2022-03-29T17:50:03.997 INFO:teuthology.orchestra.run.smithi138.stdout:30064771080 2022-03-29T17:50:04.002 INFO:tasks.ceph.ceph_manager.ceph:need seq 30064771080 got 30064771080 for osd.2 2022-03-29T17:50:04.002 DEBUG:teuthology.parallel:result is None 2022-03-29T17:50:04.012 INFO:tasks.ceph.ceph_manager.ceph:need seq 30064771080 got 30064771080 for osd.0 2022-03-29T17:50:04.012 DEBUG:teuthology.parallel:result is None 2022-03-29T17:50:04.013 INFO:tasks.ceph.ceph_manager.ceph:need seq 30064771080 got 30064771080 for osd.1 2022-03-29T17:50:04.014 DEBUG:teuthology.parallel:result is None 2022-03-29T17:50:04.014 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-03-29T17:50:04.014 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-03-29T17:50:04.235 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:04,233 [shard 0] alienstore - stat 2022-03-29T17:50:04.245 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:04,243 [shard 0] alienstore - stat 2022-03-29T17:50:04.245 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:04,244 [shard 0] alienstore - stat 2022-03-29T17:50:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:50:04.362 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2022-03-29T17:50:04.373 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2022-03-29T17:50:03.597674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":16784,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":15424,"kb_avail":314556016,"statfs":{"total":322122547200,"available":322105360384,"internally_reserved":0,"allocated":1105920,"data_stored":989039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15794176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.797452+0000","last_change":"2022-03-29T17:50:01.797452+0000","last_active":"2022-03-29T17:50:01.797452+0000","last_peered":"2022-03-29T17:50:01.797452+0000","last_clean":"2022-03-29T17:50:01.797452+0000","last_became_active":"2022-03-29T17:50:01.794743+0000","last_became_peered":"2022-03-29T17:50:01.794743+0000","last_unstale":"2022-03-29T17:50:01.797452+0000","last_undegraded":"2022-03-29T17:50:01.797452+0000","last_fullsized":"2022-03-29T17:50:01.797452+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.801372+0000","last_change":"2022-03-29T17:50:01.801372+0000","last_active":"2022-03-29T17:50:01.801372+0000","last_peered":"2022-03-29T17:50:01.801372+0000","last_clean":"2022-03-29T17:50:01.801372+0000","last_became_active":"2022-03-29T17:50:01.798893+0000","last_became_peered":"2022-03-29T17:50:01.798893+0000","last_unstale":"2022-03-29T17:50:01.801372+0000","last_undegraded":"2022-03-29T17:50:01.801372+0000","last_fullsized":"2022-03-29T17:50:01.801372+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.793368+0000","last_change":"2022-03-29T17:50:01.793368+0000","last_active":"2022-03-29T17:50:01.793368+0000","last_peered":"2022-03-29T17:50:01.793368+0000","last_clean":"2022-03-29T17:50:01.793368+0000","last_became_active":"2022-03-29T17:50:01.790143+0000","last_became_peered":"2022-03-29T17:50:01.790143+0000","last_unstale":"2022-03-29T17:50:01.793368+0000","last_undegraded":"2022-03-29T17:50:01.793368+0000","last_fullsized":"2022-03-29T17:50:01.793368+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.818717+0000","last_change":"2022-03-29T17:50:01.819321+0000","last_active":"2022-03-29T17:50:01.818717+0000","last_peered":"2022-03-29T17:50:01.818717+0000","last_clean":"2022-03-29T17:50:00.701078+0000","last_became_active":"2022-03-29T17:50:01.816613+0000","last_became_peered":"2022-03-29T17:50:01.816613+0000","last_unstale":"2022-03-29T17:50:01.818717+0000","last_undegraded":"2022-03-29T17:50:01.818717+0000","last_fullsized":"2022-03-29T17:50:01.818717+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.735228+0000","last_change":"2022-03-29T17:49:58.752345+0000","last_active":"2022-03-29T17:50:01.735228+0000","last_peered":"2022-03-29T17:50:01.735228+0000","last_clean":"2022-03-29T17:50:01.735228+0000","last_became_active":"2022-03-29T17:49:58.749708+0000","last_became_peered":"2022-03-29T17:49:58.749708+0000","last_unstale":"2022-03-29T17:50:01.735228+0000","last_undegraded":"2022-03-29T17:50:01.735228+0000","last_fullsized":"2022-03-29T17:50:01.735228+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:49:57.683718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:49:57.683718+0000","last_clean_scrub_stamp":"2022-03-29T17:49:57.683718+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.783921+0000","last_change":"2022-03-29T17:50:01.783921+0000","last_active":"2022-03-29T17:50:01.783921+0000","last_peered":"2022-03-29T17:50:01.783921+0000","last_clean":"2022-03-29T17:50:01.783921+0000","last_became_active":"2022-03-29T17:50:01.777345+0000","last_became_peered":"2022-03-29T17:50:01.777345+0000","last_unstale":"2022-03-29T17:50:01.783921+0000","last_undegraded":"2022-03-29T17:50:01.783921+0000","last_fullsized":"2022-03-29T17:50:01.783921+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.822425+0000","last_change":"2022-03-29T17:50:01.822911+0000","last_active":"2022-03-29T17:50:01.822425+0000","last_peered":"2022-03-29T17:50:01.822425+0000","last_clean":"2022-03-29T17:50:00.701078+0000","last_became_active":"2022-03-29T17:50:01.820748+0000","last_became_peered":"2022-03-29T17:50:01.820748+0000","last_unstale":"2022-03-29T17:50:01.822425+0000","last_undegraded":"2022-03-29T17:50:01.822425+0000","last_fullsized":"2022-03-29T17:50:01.822425+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.829565+0000","last_change":"2022-03-29T17:50:01.830057+0000","last_active":"2022-03-29T17:50:01.829565+0000","last_peered":"2022-03-29T17:50:01.829565+0000","last_clean":"2022-03-29T17:50:00.701078+0000","last_became_active":"2022-03-29T17:50:01.827587+0000","last_became_peered":"2022-03-29T17:50:01.827587+0000","last_unstale":"2022-03-29T17:50:01.829565+0000","last_undegraded":"2022-03-29T17:50:01.829565+0000","last_fullsized":"2022-03-29T17:50:01.829565+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.786639+0000","last_change":"2022-03-29T17:50:01.786639+0000","last_active":"2022-03-29T17:50:01.786639+0000","last_peered":"2022-03-29T17:50:01.786639+0000","last_clean":"2022-03-29T17:50:01.786639+0000","last_became_active":"2022-03-29T17:50:01.781944+0000","last_became_peered":"2022-03-29T17:50:01.781944+0000","last_unstale":"2022-03-29T17:50:01.786639+0000","last_undegraded":"2022-03-29T17:50:01.786639+0000","last_fullsized":"2022-03-29T17:50:01.786639+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":7,"seq":30064771080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4924,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852676,"statfs":{"total":107374182400,"available":107369140224,"internally_reserved":0,"allocated":53248,"data_stored":17614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":7,"seq":30064771080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5960,"kb_used_data":512,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851640,"statfs":{"total":107374182400,"available":107368079360,"internally_reserved":0,"allocated":524288,"data_stored":484570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":7,"seq":30064771080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5900,"kb_used_data":516,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851700,"statfs":{"total":107374182400,"available":107368140800,"internally_reserved":0,"allocated":528384,"data_stored":486855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2022-03-29T17:50:04.374 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-03-29T17:50:04.676 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:50:04.678 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2022-03-29T17:50:04.689 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2022-03-29T17:50:03.597674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":16784,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":15424,"kb_avail":314556016,"statfs":{"total":322122547200,"available":322105360384,"internally_reserved":0,"allocated":1105920,"data_stored":989039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15794176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.797452+0000","last_change":"2022-03-29T17:50:01.797452+0000","last_active":"2022-03-29T17:50:01.797452+0000","last_peered":"2022-03-29T17:50:01.797452+0000","last_clean":"2022-03-29T17:50:01.797452+0000","last_became_active":"2022-03-29T17:50:01.794743+0000","last_became_peered":"2022-03-29T17:50:01.794743+0000","last_unstale":"2022-03-29T17:50:01.797452+0000","last_undegraded":"2022-03-29T17:50:01.797452+0000","last_fullsized":"2022-03-29T17:50:01.797452+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.801372+0000","last_change":"2022-03-29T17:50:01.801372+0000","last_active":"2022-03-29T17:50:01.801372+0000","last_peered":"2022-03-29T17:50:01.801372+0000","last_clean":"2022-03-29T17:50:01.801372+0000","last_became_active":"2022-03-29T17:50:01.798893+0000","last_became_peered":"2022-03-29T17:50:01.798893+0000","last_unstale":"2022-03-29T17:50:01.801372+0000","last_undegraded":"2022-03-29T17:50:01.801372+0000","last_fullsized":"2022-03-29T17:50:01.801372+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.793368+0000","last_change":"2022-03-29T17:50:01.793368+0000","last_active":"2022-03-29T17:50:01.793368+0000","last_peered":"2022-03-29T17:50:01.793368+0000","last_clean":"2022-03-29T17:50:01.793368+0000","last_became_active":"2022-03-29T17:50:01.790143+0000","last_became_peered":"2022-03-29T17:50:01.790143+0000","last_unstale":"2022-03-29T17:50:01.793368+0000","last_undegraded":"2022-03-29T17:50:01.793368+0000","last_fullsized":"2022-03-29T17:50:01.793368+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.818717+0000","last_change":"2022-03-29T17:50:01.819321+0000","last_active":"2022-03-29T17:50:01.818717+0000","last_peered":"2022-03-29T17:50:01.818717+0000","last_clean":"2022-03-29T17:50:00.701078+0000","last_became_active":"2022-03-29T17:50:01.816613+0000","last_became_peered":"2022-03-29T17:50:01.816613+0000","last_unstale":"2022-03-29T17:50:01.818717+0000","last_undegraded":"2022-03-29T17:50:01.818717+0000","last_fullsized":"2022-03-29T17:50:01.818717+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.735228+0000","last_change":"2022-03-29T17:49:58.752345+0000","last_active":"2022-03-29T17:50:01.735228+0000","last_peered":"2022-03-29T17:50:01.735228+0000","last_clean":"2022-03-29T17:50:01.735228+0000","last_became_active":"2022-03-29T17:49:58.749708+0000","last_became_peered":"2022-03-29T17:49:58.749708+0000","last_unstale":"2022-03-29T17:50:01.735228+0000","last_undegraded":"2022-03-29T17:50:01.735228+0000","last_fullsized":"2022-03-29T17:50:01.735228+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:49:57.683718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:49:57.683718+0000","last_clean_scrub_stamp":"2022-03-29T17:49:57.683718+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.783921+0000","last_change":"2022-03-29T17:50:01.783921+0000","last_active":"2022-03-29T17:50:01.783921+0000","last_peered":"2022-03-29T17:50:01.783921+0000","last_clean":"2022-03-29T17:50:01.783921+0000","last_became_active":"2022-03-29T17:50:01.777345+0000","last_became_peered":"2022-03-29T17:50:01.777345+0000","last_unstale":"2022-03-29T17:50:01.783921+0000","last_undegraded":"2022-03-29T17:50:01.783921+0000","last_fullsized":"2022-03-29T17:50:01.783921+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.822425+0000","last_change":"2022-03-29T17:50:01.822911+0000","last_active":"2022-03-29T17:50:01.822425+0000","last_peered":"2022-03-29T17:50:01.822425+0000","last_clean":"2022-03-29T17:50:00.701078+0000","last_became_active":"2022-03-29T17:50:01.820748+0000","last_became_peered":"2022-03-29T17:50:01.820748+0000","last_unstale":"2022-03-29T17:50:01.822425+0000","last_undegraded":"2022-03-29T17:50:01.822425+0000","last_fullsized":"2022-03-29T17:50:01.822425+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.829565+0000","last_change":"2022-03-29T17:50:01.830057+0000","last_active":"2022-03-29T17:50:01.829565+0000","last_peered":"2022-03-29T17:50:01.829565+0000","last_clean":"2022-03-29T17:50:00.701078+0000","last_became_active":"2022-03-29T17:50:01.827587+0000","last_became_peered":"2022-03-29T17:50:01.827587+0000","last_unstale":"2022-03-29T17:50:01.829565+0000","last_undegraded":"2022-03-29T17:50:01.829565+0000","last_fullsized":"2022-03-29T17:50:01.829565+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-03-29T17:50:01.786639+0000","last_change":"2022-03-29T17:50:01.786639+0000","last_active":"2022-03-29T17:50:01.786639+0000","last_peered":"2022-03-29T17:50:01.786639+0000","last_clean":"2022-03-29T17:50:01.786639+0000","last_became_active":"2022-03-29T17:50:01.781944+0000","last_became_peered":"2022-03-29T17:50:01.781944+0000","last_unstale":"2022-03-29T17:50:01.786639+0000","last_undegraded":"2022-03-29T17:50:01.786639+0000","last_fullsized":"2022-03-29T17:50:01.786639+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-29T17:50:00.701078+0000","last_clean_scrub_stamp":"2022-03-29T17:50:00.701078+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":7,"seq":30064771080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4924,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852676,"statfs":{"total":107374182400,"available":107369140224,"internally_reserved":0,"allocated":53248,"data_stored":17614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":7,"seq":30064771080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5960,"kb_used_data":512,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851640,"statfs":{"total":107374182400,"available":107368079360,"internally_reserved":0,"allocated":524288,"data_stored":484570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":7,"seq":30064771080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5900,"kb_used_data":516,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851700,"statfs":{"total":107374182400,"available":107368140800,"internally_reserved":0,"allocated":528384,"data_stored":486855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2022-03-29T17:50:04.690 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-03-29T17:50:04.690 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-29T17:50:04.691 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-03-29T17:50:04.691 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-03-29T17:50:04.731 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:04,729 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=1189765500397987370 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:50:04.732 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:04,730 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=6606747234233674867 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:50:04.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:04,731 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=16779262909886840547 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:50:04.733 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:04,731 [shard 0] monc - get_auth_request(con=[osd.0(client) v2:172.21.15.138:6805/35577@61617 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:50:04.734 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:04,732 [shard 0] monc - get_auth_request(con=[osd.2(client) v2:172.21.15.138:6803/35579@54975 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:50:04.734 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:04,730 [shard 0] monc - get_auth_request(con=[osd.1(client) v2:172.21.15.138:6800/35578@62576 >> mgr.? v2:172.21.15.138:6812/35195], auth_method=0) 2022-03-29T17:50:04.815 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:04,813 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577@61617 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=6, pgs=27, cs=0, client_cookie=16779262909886840547, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:04.815 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:04,814 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:50:04.816 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:04,815 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578@62576 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=8, pgs=28, cs=0, client_cookie=1189765500397987370, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:04.817 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:04,815 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@54975 >> mgr.4102 v2:172.21.15.138:6812/35195] connected: gs=6, pgs=29, cs=0, client_cookie=6606747234233674867, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:04.817 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:04,816 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:50:04.818 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:04,816 [shard 0] mgrc - handle_mgr_conf mgrconfigure(period=5, threshold=5) v4 2022-03-29T17:50:05.031 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-03-29T17:50:05.032 INFO:teuthology.orchestra.run.smithi138.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-29T17:50:05.044 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-03-29T17:50:05.045 INFO:teuthology.run_tasks:Running task workunit... 2022-03-29T17:50:05.059 INFO:tasks.workunit:Pulling workunits from ref 60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 2022-03-29T17:50:05.061 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-03-29T17:50:05.061 DEBUG:teuthology.orchestra.run.smithi138:> stat -- /home/ubuntu/cephtest/mnt.0 2022-03-29T17:50:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-29T17:50:05.102 INFO:teuthology.orchestra.run.smithi138.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-03-29T17:50:05.102 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -- /home/ubuntu/cephtest/mnt.0 2022-03-29T17:50:05.159 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2022-03-29T17:50:05.160 DEBUG:teuthology.orchestra.run.smithi138:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2022-03-29T17:50:05.215 INFO:tasks.workunit:timeout=3h 2022-03-29T17:50:05.216 INFO:tasks.workunit:cleanup=True 2022-03-29T17:50:05.217 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 2022-03-29T17:50:05.238 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:05,236 [shard 0] alienstore - stat 2022-03-29T17:50:05.247 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:05,246 [shard 0] alienstore - stat 2022-03-29T17:50:05.248 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:05,247 [shard 0] alienstore - stat 2022-03-29T17:50:05.254 INFO:tasks.workunit.client.0.smithi138.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-03-29T17:50:06.244 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:06,242 [shard 0] alienstore - stat 2022-03-29T17:50:06.254 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:06,253 [shard 0] alienstore - stat 2022-03-29T17:50:06.255 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:06,254 [shard 0] alienstore - stat 2022-03-29T17:50:06.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,745 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:50:06.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,745 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:50:06.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,745 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:50:06.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,745 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 21.445096970s 2022-03-29T17:50:06.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,746 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 21.445096970s (sent 29.441825867s) 2022-03-29T17:50:06.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,746 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,746 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:50:06.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,746 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,746 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:50:06.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,746 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 21.445096970s ub 29.441825867s int 16.000000000s))): start 2022-03-29T17:50:06.751 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:06,747 [shard 0] osd - evt epoch is 13, i have 12, will wait 2022-03-29T17:50:06.751 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:06,747 [shard 0] osd - osdmap_subscribe(12) 2022-03-29T17:50:06.752 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:06,747 [shard 0] osd - handle_osd_map osd_map(13..13 src has 1..13) v4 2022-03-29T17:50:06.752 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:06,747 [shard 0] osd - handle_osd_map epochs [13..13], i have 12, src has [1..13] 2022-03-29T17:50:06.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,748 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:06.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,748 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:50:06.760 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:06,749 [shard 0] osd - osd.0: committed_osd_maps(13, 13) 2022-03-29T17:50:06.760 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:06,749 [shard 0] osd - osd.0: map e 13 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:06.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_advance_map(id=10, detail=PGAdvanceMap(pg=1.0 from=12 to=13)): start 2022-03-29T17:50:06.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_advance_map(id=11, detail=PGAdvanceMap(pg=2.2 from=12 to=13)): start 2022-03-29T17:50:06.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_advance_map(id=12, detail=PGAdvanceMap(pg=2.4 from=12 to=13)): start 2022-03-29T17:50:06.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_advance_map(id=13, detail=PGAdvanceMap(pg=2.5 from=12 to=13)): start 2022-03-29T17:50:06.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_advance_map(id=14, detail=PGAdvanceMap(pg=2.6 from=12 to=13)): start 2022-03-29T17:50:06.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_advance_map(id=15, detail=PGAdvanceMap(pg=2.7 from=12 to=13)): start 2022-03-29T17:50:06.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 12 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:06.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:50:06.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:50:06.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 12 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:06.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:50:06.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:50:06.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:50:06.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:06.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 12 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:06.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:50:06.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:50:06.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:50:06.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:06.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 12 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:06.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:50:06.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,750 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:50:06.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:50:06.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:06.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 12 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:50:06.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active state: Started advmap 2022-03-29T17:50:06.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active check_recovery_sources no source osds () went down 2022-03-29T17:50:06.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 12 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:50:06.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:50:06.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:50:06.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:06.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:50:06.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:06.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map 2022-03-29T17:50:06.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:06.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:50:06.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:50:06.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:50:06.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:06.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:9 2022-03-29T17:50:06.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:06.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_advance_map(id=12, detail=PGAdvanceMap(pg=2.4 from=12 to=13)): complete 2022-03-29T17:50:06.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_advance_map(id=13, detail=PGAdvanceMap(pg=2.5 from=12 to=13)): complete 2022-03-29T17:50:06.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_advance_map(id=15, detail=PGAdvanceMap(pg=2.7 from=12 to=13)): complete 2022-03-29T17:50:06.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_advance_map(id=10, detail=PGAdvanceMap(pg=1.0 from=12 to=13)): complete 2022-03-29T17:50:06.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_advance_map(id=14, detail=PGAdvanceMap(pg=2.6 from=12 to=13)): complete 2022-03-29T17:50:06.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_advance_map(id=11, detail=PGAdvanceMap(pg=2.2 from=12 to=13)): complete 2022-03-29T17:50:06.774 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:06,752 [shard 0] osd - osd.0: now active 2022-03-29T17:50:06.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 21.445096970s ub 29.441825867s int 16.000000000s))): got map 13 2022-03-29T17:50:06.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 21.445096970s ub 29.441825867s int 16.000000000s))): pg present 2022-03-29T17:50:06.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 21.445096970s ub 29.441825867s int 16.000000000s) for pg: 1.0 2022-03-29T17:50:06.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 21.445096970s ub 29.441825867s int 16.000000000s) 2022-03-29T17:50:06.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 29.361604691s 2022-03-29T17:50:06.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 29.361604691s 2022-03-29T17:50:06.776 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.776 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 21.445096970s ub 29.441825867s int 16.000000000s))): submitting ctx 2022-03-29T17:50:06.776 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,752 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.776 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:06,753 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 21.445096970s ub 29.441825867s int 16.000000000s))): complete 2022-03-29T17:50:06.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,787 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:50:06.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,787 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 29.441825867s))): start 2022-03-29T17:50:06.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,787 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 29.441825867s))): got map 13 2022-03-29T17:50:06.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,787 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 29.441825867s))): pg present 2022-03-29T17:50:06.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,787 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 29.441825867s) for pg: 1.0 2022-03-29T17:50:06.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,787 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 29.441825867s 2022-03-29T17:50:06.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,787 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 29.441825867s (sent 29.441825867s) 2022-03-29T17:50:06.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,788 [shard 0] osd - log is not dirty 2022-03-29T17:50:06.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,788 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 29.441825867s))): submitting ctx 2022-03-29T17:50:06.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,788 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:06.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:06,788 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 29.441825867s))): complete 2022-03-29T17:50:07.250 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:07,248 [shard 0] alienstore - stat 2022-03-29T17:50:07.259 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:07,258 [shard 0] alienstore - stat 2022-03-29T17:50:07.260 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:07,259 [shard 0] alienstore - stat 2022-03-29T17:50:08.256 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:08,255 [shard 0] alienstore - stat 2022-03-29T17:50:08.266 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:08,265 [shard 0] alienstore - stat 2022-03-29T17:50:08.267 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:08,265 [shard 0] alienstore - stat 2022-03-29T17:50:09.262 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:09,260 [shard 0] alienstore - stat 2022-03-29T17:50:09.272 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:09,270 [shard 0] alienstore - stat 2022-03-29T17:50:09.273 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:09,271 [shard 0] alienstore - stat 2022-03-29T17:50:09.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,773 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:09.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,773 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:09.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,773 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:50:09.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,773 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 24.472705841s 2022-03-29T17:50:09.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,773 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.472705841s (sent 32.469573975s) 2022-03-29T17:50:09.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,773 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,774 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:09.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,774 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,774 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:09.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,774 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.472705841s ub 32.469573975s int 16.000000000s))): start 2022-03-29T17:50:09.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:09,775 [shard 0] osd - evt epoch is 13, i have 12, will wait 2022-03-29T17:50:09.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:09,775 [shard 0] osd - osdmap_subscribe(12) 2022-03-29T17:50:09.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,775 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:09.784 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:09,776 [shard 0] osd - handle_osd_map osd_map(13..13 src has 1..13) v4 2022-03-29T17:50:09.784 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:09,776 [shard 0] osd - handle_osd_map epochs [13..13], i have 12, src has [1..13] 2022-03-29T17:50:09.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,776 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:50:09.785 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:09,778 [shard 0] osd - osd.2: committed_osd_maps(13, 13) 2022-03-29T17:50:09.785 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:09,778 [shard 0] osd - osd.2: map e 13 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:50:09.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_advance_map(id=6, detail=PGAdvanceMap(pg=2.0 from=12 to=13)): start 2022-03-29T17:50:09.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_advance_map(id=7, detail=PGAdvanceMap(pg=2.1 from=12 to=13)): start 2022-03-29T17:50:09.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_advance_map(id=8, detail=PGAdvanceMap(pg=2.3 from=12 to=13)): start 2022-03-29T17:50:09.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 12 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:50:09.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:50:09.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:50:09.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:50:09.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:09.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 12 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:50:09.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:50:09.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:50:09.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:09.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:50:09.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:50:09.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:50:09.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:09.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:9 2022-03-29T17:50:09.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:09.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,779 [shard 0] osd - pg_epoch 12 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:50:09.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:50:09.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:50:09.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:50:09.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:50:09.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:50:09.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:50:09.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:50:09.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 12:9 2022-03-29T17:50:09.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 13 2022-03-29T17:50:09.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,780 [shard 0] osd - pg_advance_map(id=8, detail=PGAdvanceMap(pg=2.3 from=12 to=13)): complete 2022-03-29T17:50:09.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,781 [shard 0] osd - pg_advance_map(id=7, detail=PGAdvanceMap(pg=2.1 from=12 to=13)): complete 2022-03-29T17:50:09.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:09.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:09.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:09.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:09.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:50:09.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 24.477348328s 2022-03-29T17:50:09.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.477348328s (sent 32.479598999s) 2022-03-29T17:50:09.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:09.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,783 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,784 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:50:09.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,784 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 24.485576630s 2022-03-29T17:50:09.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,784 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.485576630s (sent 32.480075836s) 2022-03-29T17:50:09.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,784 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,784 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:09.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,784 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,784 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:09.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,785 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:09.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,785 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.477348328s ub 32.479598999s int 16.000000000s))): start 2022-03-29T17:50:09.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,785 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:09.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,785 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.477348328s ub 32.479598999s int 16.000000000s))): got map 13 2022-03-29T17:50:09.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.477348328s ub 32.479598999s int 16.000000000s))): pg present 2022-03-29T17:50:09.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:09.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.485576630s ub 32.480075836s int 16.000000000s))): start 2022-03-29T17:50:09.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.477348328s ub 32.479598999s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:09.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.477348328s ub 32.479598999s int 16.000000000s) 2022-03-29T17:50:09.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 32.395259857s 2022-03-29T17:50:09.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 32.395259857s 2022-03-29T17:50:09.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.477348328s ub 32.479598999s int 16.000000000s))): submitting ctx 2022-03-29T17:50:09.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.485576630s ub 32.480075836s int 16.000000000s))): got map 13 2022-03-29T17:50:09.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.485576630s ub 32.480075836s int 16.000000000s))): pg present 2022-03-29T17:50:09.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,786 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.485576630s ub 32.480075836s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:09.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.485576630s ub 32.480075836s int 16.000000000s) 2022-03-29T17:50:09.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 32.395912170s 2022-03-29T17:50:09.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 32.395912170s 2022-03-29T17:50:09.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.485576630s ub 32.480075836s int 16.000000000s))): submitting ctx 2022-03-29T17:50:09.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - pg_advance_map(id=6, detail=PGAdvanceMap(pg=2.0 from=12 to=13)): complete 2022-03-29T17:50:09.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.472705841s ub 32.469573975s int 16.000000000s))): got map 13 2022-03-29T17:50:09.810 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:09,787 [shard 0] osd - osd.2: now active 2022-03-29T17:50:09.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.472705841s ub 32.469573975s int 16.000000000s))): pg present 2022-03-29T17:50:09.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.472705841s ub 32.469573975s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:09.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.472705841s ub 32.469573975s int 16.000000000s) 2022-03-29T17:50:09.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 32.420673370s 2022-03-29T17:50:09.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 32.420673370s 2022-03-29T17:50:09.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,787 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.477348328s ub 32.479598999s int 16.000000000s))): complete 2022-03-29T17:50:09.813 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,788 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.813 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,788 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.472705841s ub 32.469573975s int 16.000000000s))): submitting ctx 2022-03-29T17:50:09.813 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,788 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,788 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:09.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,788 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 32.469573975s))): start 2022-03-29T17:50:09.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 32.469573975s))): got map 13 2022-03-29T17:50:09.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 32.469573975s))): pg present 2022-03-29T17:50:09.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 32.469573975s) for pg: 2.3 2022-03-29T17:50:09.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 32.469573975s 2022-03-29T17:50:09.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.469573975s (sent 32.469573975s) 2022-03-29T17:50:09.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 32.469573975s))): submitting ctx 2022-03-29T17:50:09.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,789 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.485576630s ub 32.480075836s int 16.000000000s))): complete 2022-03-29T17:50:09.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,790 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 32.469573975s))): complete 2022-03-29T17:50:09.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,790 [shard 0] osd - peering_event(id=22, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.472705841s ub 32.469573975s int 16.000000000s))): complete 2022-03-29T17:50:09.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,793 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:09.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,793 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:09.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,793 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:09.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,793 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:09.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,793 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:50:09.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,793 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 24.494297028s 2022-03-29T17:50:09.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,793 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.494297028s (sent 32.489784241s) 2022-03-29T17:50:09.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:09.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:50:09.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 24.490119934s 2022-03-29T17:50:09.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.490119934s (sent 32.490249634s) 2022-03-29T17:50:09.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:09.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,794 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:09.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,795 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:09.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,795 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.494297028s ub 32.489784241s int 16.000000000s))): start 2022-03-29T17:50:09.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,795 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:09.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,795 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:09.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,795 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.479598999s))): start 2022-03-29T17:50:09.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,795 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.494297028s ub 32.489784241s int 16.000000000s))): got map 13 2022-03-29T17:50:09.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,796 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.494297028s ub 32.489784241s int 16.000000000s))): pg present 2022-03-29T17:50:09.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,796 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:09.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,796 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.490119934s ub 32.490249634s int 16.000000000s))): start 2022-03-29T17:50:09.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,796 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.479598999s))): got map 13 2022-03-29T17:50:09.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,796 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:09.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,796 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.480075836s))): start 2022-03-29T17:50:09.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,796 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.479598999s))): pg present 2022-03-29T17:50:09.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,796 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.494297028s ub 32.489784241s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:09.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.494297028s ub 32.489784241s int 16.000000000s) 2022-03-29T17:50:09.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 32.405933380s 2022-03-29T17:50:09.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 32.405933380s 2022-03-29T17:50:09.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.494297028s ub 32.489784241s int 16.000000000s))): submitting ctx 2022-03-29T17:50:09.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.480075836s))): got map 13 2022-03-29T17:50:09.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.479598999s) for pg: 2.4 2022-03-29T17:50:09.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 32.479598999s 2022-03-29T17:50:09.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.479598999s (sent 32.479598999s) 2022-03-29T17:50:09.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.479598999s))): submitting ctx 2022-03-29T17:50:09.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.490119934s ub 32.490249634s int 16.000000000s))): got map 13 2022-03-29T17:50:09.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,797 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.490119934s ub 32.490249634s int 16.000000000s))): pg present 2022-03-29T17:50:09.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.490119934s ub 32.490249634s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:09.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.490119934s ub 32.490249634s int 16.000000000s) 2022-03-29T17:50:09.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 32.407169342s 2022-03-29T17:50:09.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 32.407169342s 2022-03-29T17:50:09.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.480075836s))): pg present 2022-03-29T17:50:09.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.480075836s) for pg: 2.5 2022-03-29T17:50:09.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 32.480075836s 2022-03-29T17:50:09.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.480075836s (sent 32.480075836s) 2022-03-29T17:50:09.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.480075836s))): submitting ctx 2022-03-29T17:50:09.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.490119934s ub 32.490249634s int 16.000000000s))): submitting ctx 2022-03-29T17:50:09.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,798 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,799 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.494297028s ub 32.489784241s int 16.000000000s))): complete 2022-03-29T17:50:09.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,799 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.479598999s))): complete 2022-03-29T17:50:09.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,799 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:09.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,799 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.489784241s))): start 2022-03-29T17:50:09.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,799 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 24.490119934s ub 32.490249634s int 16.000000000s))): complete 2022-03-29T17:50:09.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,800 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.489784241s))): got map 13 2022-03-29T17:50:09.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,800 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.489784241s))): pg present 2022-03-29T17:50:09.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,800 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.480075836s))): complete 2022-03-29T17:50:09.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,800 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:09.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,800 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.490249634s))): start 2022-03-29T17:50:09.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.489784241s) for pg: 2.6 2022-03-29T17:50:09.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 32.489784241s 2022-03-29T17:50:09.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.489784241s (sent 32.489784241s) 2022-03-29T17:50:09.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.489784241s))): submitting ctx 2022-03-29T17:50:09.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.490249634s))): got map 13 2022-03-29T17:50:09.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.490249634s))): pg present 2022-03-29T17:50:09.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.490249634s) for pg: 2.7 2022-03-29T17:50:09.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,801 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 32.490249634s 2022-03-29T17:50:09.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,802 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.490249634s (sent 32.490249634s) 2022-03-29T17:50:09.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,802 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,802 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.490249634s))): submitting ctx 2022-03-29T17:50:09.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,802 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,802 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.489784241s))): complete 2022-03-29T17:50:09.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,802 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 32.490249634s))): complete 2022-03-29T17:50:09.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,807 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:09.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,807 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:09.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,807 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:50:09.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,807 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 24.449274063s 2022-03-29T17:50:09.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,807 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.449274063s (sent 32.440673828s) 2022-03-29T17:50:09.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,807 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.854 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,807 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:09.854 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,807 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,808 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:09.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,808 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.449274063s ub 32.440673828s int 16.000000000s))): start 2022-03-29T17:50:09.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,808 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.449274063s ub 32.440673828s int 16.000000000s))): got map 13 2022-03-29T17:50:09.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,808 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.449274063s ub 32.440673828s int 16.000000000s))): pg present 2022-03-29T17:50:09.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,808 [shard 0] osd - peering_event(id=23, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:09.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,809 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.449274063s ub 32.440673828s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:09.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,809 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.449274063s ub 32.440673828s int 16.000000000s) 2022-03-29T17:50:09.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,809 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 32.505264282s 2022-03-29T17:50:09.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,809 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 32.505264282s 2022-03-29T17:50:09.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,809 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,809 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.449274063s ub 32.440673828s int 16.000000000s))): submitting ctx 2022-03-29T17:50:09.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,809 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,810 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.449274063s ub 32.440673828s int 16.000000000s))): complete 2022-03-29T17:50:09.861 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,817 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:09.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,817 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:09.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,817 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:50:09.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,817 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 24.453264236s 2022-03-29T17:50:09.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,817 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.453264236s (sent 32.450931549s) 2022-03-29T17:50:09.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:09.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:09.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.440673828s))): start 2022-03-29T17:50:09.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - peering_event(id=24, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:09.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.440673828s))): got map 13 2022-03-29T17:50:09.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.440673828s))): pg present 2022-03-29T17:50:09.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.440673828s) for pg: 2.0 2022-03-29T17:50:09.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 32.440673828s 2022-03-29T17:50:09.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,818 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.440673828s (sent 32.440673828s) 2022-03-29T17:50:09.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,819 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,819 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.440673828s))): submitting ctx 2022-03-29T17:50:09.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,819 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,819 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:09.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,819 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.453264236s ub 32.450931549s int 16.000000000s))): start 2022-03-29T17:50:09.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,819 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.453264236s ub 32.450931549s int 16.000000000s))): got map 13 2022-03-29T17:50:09.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,820 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.453264236s ub 32.450931549s int 16.000000000s))): pg present 2022-03-29T17:50:09.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,820 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.453264236s ub 32.450931549s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:09.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,820 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.453264236s ub 32.450931549s int 16.000000000s) 2022-03-29T17:50:09.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,820 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 32.516422272s 2022-03-29T17:50:09.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,820 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 32.516422272s 2022-03-29T17:50:09.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,820 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,820 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.453264236s ub 32.450931549s int 16.000000000s))): submitting ctx 2022-03-29T17:50:09.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,820 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,821 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 24.453264236s ub 32.450931549s int 16.000000000s))): complete 2022-03-29T17:50:09.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,821 [shard 0] osd - peering_event(id=25, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.440673828s))): complete 2022-03-29T17:50:09.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,829 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:09.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,829 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:09.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,829 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:50:09.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,829 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 24.436038971s 2022-03-29T17:50:09.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,829 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 24.436038971s (sent 32.438499451s) 2022-03-29T17:50:09.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,829 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,829 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:09.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,829 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,830 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:09.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,830 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 24.436038971s ub 32.438499451s int 16.000000000s))): start 2022-03-29T17:50:09.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,830 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 24.436038971s ub 32.438499451s int 16.000000000s))): got map 13 2022-03-29T17:50:09.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,830 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 24.436038971s ub 32.438499451s int 16.000000000s))): pg present 2022-03-29T17:50:09.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,830 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:09.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,831 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 24.436038971s ub 32.438499451s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:09.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,831 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 24.436038971s ub 32.438499451s int 16.000000000s) 2022-03-29T17:50:09.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,831 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 32.527271271s 2022-03-29T17:50:09.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,831 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 32.527271271s 2022-03-29T17:50:09.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,831 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,831 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 24.436038971s ub 32.438499451s int 16.000000000s))): submitting ctx 2022-03-29T17:50:09.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,831 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:09,832 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 24.436038971s ub 32.438499451s int 16.000000000s))): complete 2022-03-29T17:50:09.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,860 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:09.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,860 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.450931549s))): start 2022-03-29T17:50:09.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,860 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.450931549s))): got map 13 2022-03-29T17:50:09.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,860 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.450931549s))): pg present 2022-03-29T17:50:09.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,860 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.450931549s) for pg: 2.1 2022-03-29T17:50:09.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,860 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 32.450931549s 2022-03-29T17:50:09.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,861 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.450931549s (sent 32.450931549s) 2022-03-29T17:50:09.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,861 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,861 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.450931549s))): submitting ctx 2022-03-29T17:50:09.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,861 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:09,861 [shard 0] osd - peering_event(id=26, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.450931549s))): complete 2022-03-29T17:50:09.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:09.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.438499451s))): start 2022-03-29T17:50:09.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.438499451s))): got map 13 2022-03-29T17:50:09.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.438499451s))): pg present 2022-03-29T17:50:09.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.438499451s) for pg: 2.2 2022-03-29T17:50:09.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 32.438499451s 2022-03-29T17:50:09.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.438499451s (sent 32.438499451s) 2022-03-29T17:50:09.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - log is not dirty 2022-03-29T17:50:09.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.438499451s))): submitting ctx 2022-03-29T17:50:09.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,871 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:09.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:09,872 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 32.438499451s))): complete 2022-03-29T17:50:10.263 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:10,261 [shard 0] alienstore - stat 2022-03-29T17:50:10.276 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:10,275 [shard 0] alienstore - stat 2022-03-29T17:50:10.280 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:10,279 [shard 0] alienstore - stat 2022-03-29T17:50:11.268 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:11,266 [shard 0] alienstore - stat 2022-03-29T17:50:11.281 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:11,280 [shard 0] alienstore - stat 2022-03-29T17:50:11.285 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:11,284 [shard 0] alienstore - stat 2022-03-29T17:50:12.272 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:12,271 [shard 0] alienstore - stat 2022-03-29T17:50:12.285 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:12,284 [shard 0] alienstore - stat 2022-03-29T17:50:12.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:12,289 [shard 0] alienstore - stat 2022-03-29T17:50:13.277 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:13,276 [shard 0] alienstore - stat 2022-03-29T17:50:13.291 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:13,289 [shard 0] alienstore - stat 2022-03-29T17:50:13.295 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:13,294 [shard 0] alienstore - stat 2022-03-29T17:50:14.282 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:14,281 [shard 0] alienstore - stat 2022-03-29T17:50:14.295 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:14,294 [shard 0] alienstore - stat 2022-03-29T17:50:14.300 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:14,299 [shard 0] alienstore - stat 2022-03-29T17:50:14.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,746 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:50:14.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,746 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:50:14.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,747 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:50:14.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,747 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 29.441825867s 2022-03-29T17:50:14.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,747 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 29.441825867s (sent 37.443000793s) 2022-03-29T17:50:14.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,747 [shard 0] osd - log is not dirty 2022-03-29T17:50:14.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,747 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:50:14.751 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,747 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:14.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:50:14.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 29.441825867s ub 37.443000793s int 16.000000000s))): start 2022-03-29T17:50:14.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 29.441825867s ub 37.443000793s int 16.000000000s))): got map 13 2022-03-29T17:50:14.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 29.441825867s ub 37.443000793s int 16.000000000s))): pg present 2022-03-29T17:50:14.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 29.441825867s ub 37.443000793s int 16.000000000s) for pg: 1.0 2022-03-29T17:50:14.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 29.441825867s ub 37.443000793s int 16.000000000s) 2022-03-29T17:50:14.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 37.357440948s 2022-03-29T17:50:14.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 37.357440948s 2022-03-29T17:50:14.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - log is not dirty 2022-03-29T17:50:14.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 29.441825867s ub 37.443000793s int 16.000000000s))): submitting ctx 2022-03-29T17:50:14.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,748 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:14.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,749 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:50:14.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,749 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:50:14.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,749 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 37.443000793s))): start 2022-03-29T17:50:14.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:14,749 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 29.441825867s ub 37.443000793s int 16.000000000s))): complete 2022-03-29T17:50:14.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 37.443000793s))): got map 13 2022-03-29T17:50:14.758 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 37.443000793s))): pg present 2022-03-29T17:50:14.758 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 37.443000793s) for pg: 1.0 2022-03-29T17:50:14.758 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 37.443000793s 2022-03-29T17:50:14.758 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 37.443000793s (sent 37.443000793s) 2022-03-29T17:50:14.759 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - log is not dirty 2022-03-29T17:50:14.759 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 37.443000793s))): submitting ctx 2022-03-29T17:50:14.759 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:14.760 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:14,750 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 37.443000793s))): complete 2022-03-29T17:50:15.287 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:15,286 [shard 0] alienstore - stat 2022-03-29T17:50:15.301 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:15,299 [shard 0] alienstore - stat 2022-03-29T17:50:15.305 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:15,304 [shard 0] alienstore - stat 2022-03-29T17:50:16.292 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:16,290 [shard 0] alienstore - stat 2022-03-29T17:50:16.306 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:16,304 [shard 0] alienstore - stat 2022-03-29T17:50:16.309 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:16,308 [shard 0] alienstore - stat 2022-03-29T17:50:17.296 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:17,295 [shard 0] alienstore - stat 2022-03-29T17:50:17.310 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:17,309 [shard 0] alienstore - stat 2022-03-29T17:50:17.313 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:17,312 [shard 0] alienstore - stat 2022-03-29T17:50:17.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,781 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:17.783 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,781 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:17.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,781 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:50:17.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,781 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 32.469573975s 2022-03-29T17:50:17.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,781 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.469573975s (sent 40.477180481s) 2022-03-29T17:50:17.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,781 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,781 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:17.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,781 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:17.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.469573975s ub 40.477180481s int 16.000000000s))): start 2022-03-29T17:50:17.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.469573975s ub 40.477180481s int 16.000000000s))): got map 13 2022-03-29T17:50:17.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.469573975s ub 40.477180481s int 16.000000000s))): pg present 2022-03-29T17:50:17.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.469573975s ub 40.477180481s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:17.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 32.469573975s ub 40.477180481s int 16.000000000s) 2022-03-29T17:50:17.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 40.415710449s 2022-03-29T17:50:17.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 40.415710449s 2022-03-29T17:50:17.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.469573975s ub 40.477180481s int 16.000000000s))): submitting ctx 2022-03-29T17:50:17.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,782 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,783 [shard 0] osd - peering_event(id=27, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.469573975s ub 40.477180481s int 16.000000000s))): complete 2022-03-29T17:50:17.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,783 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:17.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,783 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:17.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,783 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 40.477180481s))): start 2022-03-29T17:50:17.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,784 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 40.477180481s))): got map 13 2022-03-29T17:50:17.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,784 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 40.477180481s))): pg present 2022-03-29T17:50:17.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,784 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 40.477180481s) for pg: 2.3 2022-03-29T17:50:17.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,784 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 40.477180481s 2022-03-29T17:50:17.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,784 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.477180481s (sent 40.477180481s) 2022-03-29T17:50:17.793 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,784 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.793 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,784 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 40.477180481s))): submitting ctx 2022-03-29T17:50:17.793 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,784 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.793 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,785 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 40.477180481s))): complete 2022-03-29T17:50:17.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:17.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:17.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:17.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:17.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:50:17.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 32.480075836s 2022-03-29T17:50:17.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.480075836s (sent 40.487148285s) 2022-03-29T17:50:17.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:17.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:50:17.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:17.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.480075836s ub 40.487148285s int 16.000000000s))): start 2022-03-29T17:50:17.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.480075836s ub 40.487148285s int 16.000000000s))): got map 13 2022-03-29T17:50:17.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,791 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.480075836s ub 40.487148285s int 16.000000000s))): pg present 2022-03-29T17:50:17.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.480075836s ub 40.487148285s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:17.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 32.480075836s ub 40.487148285s int 16.000000000s) 2022-03-29T17:50:17.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 40.401004791s 2022-03-29T17:50:17.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 40.401004791s 2022-03-29T17:50:17.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.480075836s ub 40.487148285s int 16.000000000s))): submitting ctx 2022-03-29T17:50:17.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 32.479598999s 2022-03-29T17:50:17.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.479598999s (sent 40.487411499s) 2022-03-29T17:50:17.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:17.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,792 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,793 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.480075836s ub 40.487148285s int 16.000000000s))): complete 2022-03-29T17:50:17.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,793 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:17.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,793 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.479598999s ub 40.487411499s int 16.000000000s))): start 2022-03-29T17:50:17.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,793 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.479598999s ub 40.487411499s int 16.000000000s))): got map 13 2022-03-29T17:50:17.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,793 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.479598999s ub 40.487411499s int 16.000000000s))): pg present 2022-03-29T17:50:17.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,793 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.479598999s ub 40.487411499s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:17.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 32.479598999s ub 40.487411499s int 16.000000000s) 2022-03-29T17:50:17.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 40.402919769s 2022-03-29T17:50:17.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 40.402919769s 2022-03-29T17:50:17.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.479598999s ub 40.487411499s int 16.000000000s))): submitting ctx 2022-03-29T17:50:17.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.479598999s ub 40.487411499s int 16.000000000s))): complete 2022-03-29T17:50:17.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:17.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,794 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:17.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:17.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487148285s))): start 2022-03-29T17:50:17.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487148285s))): got map 13 2022-03-29T17:50:17.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487148285s))): pg present 2022-03-29T17:50:17.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487148285s) for pg: 2.5 2022-03-29T17:50:17.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 40.487148285s 2022-03-29T17:50:17.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.487148285s (sent 40.487148285s) 2022-03-29T17:50:17.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,795 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487148285s))): submitting ctx 2022-03-29T17:50:17.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,796 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,796 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:17.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,796 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487411499s))): start 2022-03-29T17:50:17.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,796 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487411499s))): got map 13 2022-03-29T17:50:17.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,796 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487411499s))): pg present 2022-03-29T17:50:17.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,796 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487148285s))): complete 2022-03-29T17:50:17.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,796 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487411499s) for pg: 2.4 2022-03-29T17:50:17.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,797 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 40.487411499s 2022-03-29T17:50:17.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,797 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.487411499s (sent 40.487411499s) 2022-03-29T17:50:17.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,797 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,797 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487411499s))): submitting ctx 2022-03-29T17:50:17.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,797 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,797 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.487411499s))): complete 2022-03-29T17:50:17.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:17.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:17.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:17.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:17.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:50:17.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 32.490249634s 2022-03-29T17:50:17.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.490249634s (sent 40.497299194s) 2022-03-29T17:50:17.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:17.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:50:17.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 32.489784241s 2022-03-29T17:50:17.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.489784241s (sent 40.497524261s) 2022-03-29T17:50:17.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:17.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,801 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,802 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:17.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,802 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:17.822 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,814 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:17.822 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,814 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:17.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,814 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:50:17.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,814 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 32.440673828s 2022-03-29T17:50:17.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,814 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.440673828s (sent 40.447814941s) 2022-03-29T17:50:17.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,814 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,814 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:17.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,814 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,815 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:17.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,815 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.440673828s ub 40.447814941s int 16.000000000s))): start 2022-03-29T17:50:17.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,815 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.440673828s ub 40.447814941s int 16.000000000s))): got map 13 2022-03-29T17:50:17.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,815 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.440673828s ub 40.447814941s int 16.000000000s))): pg present 2022-03-29T17:50:17.826 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,815 [shard 0] osd - peering_event(id=28, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:17.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,816 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.440673828s ub 40.447814941s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:17.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,816 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 32.440673828s ub 40.447814941s int 16.000000000s) 2022-03-29T17:50:17.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,816 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 40.512332916s 2022-03-29T17:50:17.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,816 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 40.512332916s 2022-03-29T17:50:17.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,816 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,816 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.440673828s ub 40.447814941s int 16.000000000s))): submitting ctx 2022-03-29T17:50:17.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,816 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,817 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.440673828s ub 40.447814941s int 16.000000000s))): complete 2022-03-29T17:50:17.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,824 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:17.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,824 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:17.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,824 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:50:17.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,824 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 32.450931549s 2022-03-29T17:50:17.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,824 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.450931549s (sent 40.457794189s) 2022-03-29T17:50:17.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,824 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,824 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:17.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,824 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:17.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.447814941s))): start 2022-03-29T17:50:17.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.447814941s))): got map 13 2022-03-29T17:50:17.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.447814941s))): pg present 2022-03-29T17:50:17.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - peering_event(id=29, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:17.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.447814941s) for pg: 2.0 2022-03-29T17:50:17.834 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 40.447814941s 2022-03-29T17:50:17.834 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.447814941s (sent 40.447814941s) 2022-03-29T17:50:17.834 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.834 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.447814941s))): submitting ctx 2022-03-29T17:50:17.834 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,825 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,826 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:17.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,826 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.450931549s ub 40.457794189s int 16.000000000s))): start 2022-03-29T17:50:17.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,826 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.450931549s ub 40.457794189s int 16.000000000s))): got map 13 2022-03-29T17:50:17.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,826 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.450931549s ub 40.457794189s int 16.000000000s))): pg present 2022-03-29T17:50:17.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,826 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.450931549s ub 40.457794189s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:17.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,826 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 32.450931549s ub 40.457794189s int 16.000000000s) 2022-03-29T17:50:17.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,826 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 40.522838593s 2022-03-29T17:50:17.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,826 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 40.522838593s 2022-03-29T17:50:17.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,827 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,827 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.450931549s ub 40.457794189s int 16.000000000s))): submitting ctx 2022-03-29T17:50:17.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,827 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,827 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 32.450931549s ub 40.457794189s int 16.000000000s))): complete 2022-03-29T17:50:17.839 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,828 [shard 0] osd - peering_event(id=30, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.447814941s))): complete 2022-03-29T17:50:17.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,837 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:17.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,837 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:17.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,837 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:17.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,837 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.490249634s ub 40.497299194s int 16.000000000s))): start 2022-03-29T17:50:17.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,837 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.490249634s ub 40.497299194s int 16.000000000s))): got map 13 2022-03-29T17:50:17.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,837 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.490249634s ub 40.497299194s int 16.000000000s))): pg present 2022-03-29T17:50:17.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,837 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:50:17.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,837 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 32.438499451s 2022-03-29T17:50:17.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 32.438499451s (sent 40.446838379s) 2022-03-29T17:50:17.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:17.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:17.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.489784241s ub 40.497524261s int 16.000000000s))): start 2022-03-29T17:50:17.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.490249634s ub 40.497299194s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:17.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 32.490249634s ub 40.497299194s int 16.000000000s) 2022-03-29T17:50:17.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 40.447479248s 2022-03-29T17:50:17.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 40.447479248s 2022-03-29T17:50:17.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.490249634s ub 40.497299194s int 16.000000000s))): submitting ctx 2022-03-29T17:50:17.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:17.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,838 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.489784241s ub 40.497524261s int 16.000000000s))): got map 13 2022-03-29T17:50:17.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.489784241s ub 40.497524261s int 16.000000000s))): pg present 2022-03-29T17:50:17.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.489784241s ub 40.497524261s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:17.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 32.489784241s ub 40.497524261s int 16.000000000s) 2022-03-29T17:50:17.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 40.448020935s 2022-03-29T17:50:17.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 40.448020935s 2022-03-29T17:50:17.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.489784241s ub 40.497524261s int 16.000000000s))): submitting ctx 2022-03-29T17:50:17.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,839 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.490249634s ub 40.497299194s int 16.000000000s))): complete 2022-03-29T17:50:17.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,840 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 32.489784241s ub 40.497524261s int 16.000000000s))): complete 2022-03-29T17:50:17.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:17.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.457794189s))): start 2022-03-29T17:50:17.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.457794189s))): got map 13 2022-03-29T17:50:17.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.457794189s))): pg present 2022-03-29T17:50:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.457794189s) for pg: 2.1 2022-03-29T17:50:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 40.457794189s 2022-03-29T17:50:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.457794189s (sent 40.457794189s) 2022-03-29T17:50:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.457794189s))): submitting ctx 2022-03-29T17:50:17.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,867 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:17,868 [shard 0] osd - peering_event(id=31, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.457794189s))): complete 2022-03-29T17:50:17.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,878 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:17.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,878 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 32.438499451s ub 40.446838379s int 16.000000000s))): start 2022-03-29T17:50:17.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,878 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 32.438499451s ub 40.446838379s int 16.000000000s))): got map 13 2022-03-29T17:50:17.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 32.438499451s ub 40.446838379s int 16.000000000s))): pg present 2022-03-29T17:50:17.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 32.438499451s ub 40.446838379s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:17.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 32.438499451s ub 40.446838379s int 16.000000000s) 2022-03-29T17:50:17.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 40.575256348s 2022-03-29T17:50:17.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 40.575256348s 2022-03-29T17:50:17.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 32.438499451s ub 40.446838379s int 16.000000000s))): submitting ctx 2022-03-29T17:50:17.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:17.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497299194s))): start 2022-03-29T17:50:17.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,879 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497299194s))): got map 13 2022-03-29T17:50:17.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497299194s))): pg present 2022-03-29T17:50:17.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:17.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497524261s))): start 2022-03-29T17:50:17.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 32.438499451s ub 40.446838379s int 16.000000000s))): complete 2022-03-29T17:50:17.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497299194s) for pg: 2.7 2022-03-29T17:50:17.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 40.497299194s 2022-03-29T17:50:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.497299194s (sent 40.497299194s) 2022-03-29T17:50:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497299194s))): submitting ctx 2022-03-29T17:50:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497524261s))): got map 13 2022-03-29T17:50:17.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497524261s))): pg present 2022-03-29T17:50:17.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497524261s) for pg: 2.6 2022-03-29T17:50:17.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 40.497524261s 2022-03-29T17:50:17.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.497524261s (sent 40.497524261s) 2022-03-29T17:50:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497524261s))): submitting ctx 2022-03-29T17:50:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,880 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497299194s))): complete 2022-03-29T17:50:17.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:17,883 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 40.497524261s))): complete 2022-03-29T17:50:17.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,919 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:17.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,919 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.446838379s))): start 2022-03-29T17:50:17.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,919 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.446838379s))): got map 13 2022-03-29T17:50:17.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,919 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.446838379s))): pg present 2022-03-29T17:50:17.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,920 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.446838379s) for pg: 2.2 2022-03-29T17:50:17.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,920 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 40.446838379s 2022-03-29T17:50:17.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,920 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.446838379s (sent 40.446838379s) 2022-03-29T17:50:17.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,920 [shard 0] osd - log is not dirty 2022-03-29T17:50:17.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,920 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.446838379s))): submitting ctx 2022-03-29T17:50:17.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,920 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:17.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:17,921 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 40.446838379s))): complete 2022-03-29T17:50:18.302 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:18,300 [shard 0] alienstore - stat 2022-03-29T17:50:18.314 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:18,313 [shard 0] alienstore - stat 2022-03-29T17:50:18.318 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:18,316 [shard 0] alienstore - stat 2022-03-29T17:50:19.305 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:19,304 [shard 0] alienstore - stat 2022-03-29T17:50:19.317 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:19,316 [shard 0] alienstore - stat 2022-03-29T17:50:19.321 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:19,320 [shard 0] alienstore - stat 2022-03-29T17:50:20.310 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:20,308 [shard 0] alienstore - stat 2022-03-29T17:50:20.322 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:20,320 [shard 0] alienstore - stat 2022-03-29T17:50:20.325 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:20,324 [shard 0] alienstore - stat 2022-03-29T17:50:21.315 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:21,313 [shard 0] alienstore - stat 2022-03-29T17:50:21.327 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:21,325 [shard 0] alienstore - stat 2022-03-29T17:50:21.330 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:21,329 [shard 0] alienstore - stat 2022-03-29T17:50:22.320 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:22,318 [shard 0] alienstore - stat 2022-03-29T17:50:22.332 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:22,330 [shard 0] alienstore - stat 2022-03-29T17:50:22.335 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:22,334 [shard 0] alienstore - stat 2022-03-29T17:50:22.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,753 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:50:22.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,753 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:50:22.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,753 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:50:22.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,753 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 37.443000793s 2022-03-29T17:50:22.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,753 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 37.443000793s (sent 45.449344635s) 2022-03-29T17:50:22.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,753 [shard 0] osd - log is not dirty 2022-03-29T17:50:22.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,753 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:50:22.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,753 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:22.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:50:22.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 37.443000793s ub 45.449344635s int 16.000000000s))): start 2022-03-29T17:50:22.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 37.443000793s ub 45.449344635s int 16.000000000s))): got map 13 2022-03-29T17:50:22.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 37.443000793s ub 45.449344635s int 16.000000000s))): pg present 2022-03-29T17:50:22.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 37.443000793s ub 45.449344635s int 16.000000000s) for pg: 1.0 2022-03-29T17:50:22.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 37.443000793s ub 45.449344635s int 16.000000000s) 2022-03-29T17:50:22.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 45.363716125s 2022-03-29T17:50:22.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 45.363716125s 2022-03-29T17:50:22.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - log is not dirty 2022-03-29T17:50:22.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 37.443000793s ub 45.449344635s int 16.000000000s))): submitting ctx 2022-03-29T17:50:22.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,754 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:22.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,755 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:50:22.763 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,755 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:50:22.763 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,755 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 45.449344635s))): start 2022-03-29T17:50:22.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:22,756 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 37.443000793s ub 45.449344635s int 16.000000000s))): complete 2022-03-29T17:50:22.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,756 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 45.449344635s))): got map 13 2022-03-29T17:50:22.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,756 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 45.449344635s))): pg present 2022-03-29T17:50:22.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,756 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 45.449344635s) for pg: 1.0 2022-03-29T17:50:22.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,756 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 45.449344635s 2022-03-29T17:50:22.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,756 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 45.449344635s (sent 45.449344635s) 2022-03-29T17:50:22.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,757 [shard 0] osd - log is not dirty 2022-03-29T17:50:22.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,757 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 45.449344635s))): submitting ctx 2022-03-29T17:50:22.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,757 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:22.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:22,757 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 45.449344635s))): complete 2022-03-29T17:50:23.327 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:23,325 [shard 0] alienstore - stat 2022-03-29T17:50:23.338 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:23,337 [shard 0] alienstore - stat 2022-03-29T17:50:23.342 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:23,341 [shard 0] alienstore - stat 2022-03-29T17:50:24.334 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:24,332 [shard 0] alienstore - stat 2022-03-29T17:50:24.347 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:24,345 [shard 0] alienstore - stat 2022-03-29T17:50:24.349 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:24,348 [shard 0] alienstore - stat 2022-03-29T17:50:25.341 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:25,339 [shard 0] alienstore - stat 2022-03-29T17:50:25.354 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:25,353 [shard 0] alienstore - stat 2022-03-29T17:50:25.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:25,354 [shard 0] alienstore - stat 2022-03-29T17:50:25.789 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,785 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:25.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,786 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:25.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,786 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:50:25.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,786 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 40.477180481s 2022-03-29T17:50:25.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,786 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.477180481s (sent 48.482231140s) 2022-03-29T17:50:25.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,786 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,786 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:25.792 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,786 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,787 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:25.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,787 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.477180481s ub 48.482231140s int 16.000000000s))): start 2022-03-29T17:50:25.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,787 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.477180481s ub 48.482231140s int 16.000000000s))): got map 13 2022-03-29T17:50:25.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,787 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.477180481s ub 48.482231140s int 16.000000000s))): pg present 2022-03-29T17:50:25.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,787 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.477180481s ub 48.482231140s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:25.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,787 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 40.477180481s ub 48.482231140s int 16.000000000s) 2022-03-29T17:50:25.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,787 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 48.420917511s 2022-03-29T17:50:25.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,787 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 48.420917511s 2022-03-29T17:50:25.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,788 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,788 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.477180481s ub 48.482231140s int 16.000000000s))): submitting ctx 2022-03-29T17:50:25.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,788 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,788 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:25.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,788 [shard 0] osd - peering_event(id=32, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.477180481s ub 48.482231140s int 16.000000000s))): complete 2022-03-29T17:50:25.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,789 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:25.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,789 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 48.482231140s))): start 2022-03-29T17:50:25.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,789 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 48.482231140s))): got map 13 2022-03-29T17:50:25.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,789 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 48.482231140s))): pg present 2022-03-29T17:50:25.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,789 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 48.482231140s) for pg: 2.3 2022-03-29T17:50:25.810 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,790 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 48.482231140s 2022-03-29T17:50:25.810 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,790 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.482231140s (sent 48.482231140s) 2022-03-29T17:50:25.811 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,790 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.811 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,790 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 48.482231140s))): submitting ctx 2022-03-29T17:50:25.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,790 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,791 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 48.482231140s))): complete 2022-03-29T17:50:25.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,795 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:25.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,795 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:25.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:25.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:25.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:50:25.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 40.487148285s 2022-03-29T17:50:25.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.487148285s (sent 48.492118835s) 2022-03-29T17:50:25.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:25.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:50:25.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 40.487411499s 2022-03-29T17:50:25.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.487411499s (sent 48.492549896s) 2022-03-29T17:50:25.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,797 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:25.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,797 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487148285s ub 48.492118835s int 16.000000000s))): start 2022-03-29T17:50:25.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,797 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487148285s ub 48.492118835s int 16.000000000s))): got map 13 2022-03-29T17:50:25.821 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,797 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487148285s ub 48.492118835s int 16.000000000s))): pg present 2022-03-29T17:50:25.821 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,797 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487148285s ub 48.492118835s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:25.821 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,797 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 40.487148285s ub 48.492118835s int 16.000000000s) 2022-03-29T17:50:25.821 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,797 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 48.406803131s 2022-03-29T17:50:25.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,797 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 48.406803131s 2022-03-29T17:50:25.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,798 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,798 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487148285s ub 48.492118835s int 16.000000000s))): submitting ctx 2022-03-29T17:50:25.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,798 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,796 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,798 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:25.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,798 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,799 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487148285s ub 48.492118835s int 16.000000000s))): complete 2022-03-29T17:50:25.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,799 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:25.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,799 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487411499s ub 48.492549896s int 16.000000000s))): start 2022-03-29T17:50:25.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,799 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487411499s ub 48.492549896s int 16.000000000s))): got map 13 2022-03-29T17:50:25.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,799 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487411499s ub 48.492549896s int 16.000000000s))): pg present 2022-03-29T17:50:25.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,799 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487411499s ub 48.492549896s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:25.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,799 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 40.487411499s ub 48.492549896s int 16.000000000s) 2022-03-29T17:50:25.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,799 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 48.408813477s 2022-03-29T17:50:25.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,800 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 48.408813477s 2022-03-29T17:50:25.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,800 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,800 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487411499s ub 48.492549896s int 16.000000000s))): submitting ctx 2022-03-29T17:50:25.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,800 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,800 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:25.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,800 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:25.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,800 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.487411499s ub 48.492549896s int 16.000000000s))): complete 2022-03-29T17:50:25.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,801 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:25.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,801 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492118835s))): start 2022-03-29T17:50:25.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,801 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492118835s))): got map 13 2022-03-29T17:50:25.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,801 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492118835s))): pg present 2022-03-29T17:50:25.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,802 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492118835s) for pg: 2.5 2022-03-29T17:50:25.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,802 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 48.492118835s 2022-03-29T17:50:25.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,802 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.492118835s (sent 48.492118835s) 2022-03-29T17:50:25.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,802 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,802 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492118835s))): submitting ctx 2022-03-29T17:50:25.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,802 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,802 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:25.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,802 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492549896s))): start 2022-03-29T17:50:25.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,803 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492549896s))): got map 13 2022-03-29T17:50:25.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,803 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492549896s))): pg present 2022-03-29T17:50:25.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,803 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492118835s))): complete 2022-03-29T17:50:25.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,803 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492549896s) for pg: 2.4 2022-03-29T17:50:25.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,803 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 48.492549896s 2022-03-29T17:50:25.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,804 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.492549896s (sent 48.492549896s) 2022-03-29T17:50:25.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,804 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,804 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492549896s))): submitting ctx 2022-03-29T17:50:25.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,804 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,805 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.492549896s))): complete 2022-03-29T17:50:25.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,805 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:25.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,805 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:25.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:25.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:25.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:50:25.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 40.497524261s 2022-03-29T17:50:25.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.497524261s (sent 48.502147675s) 2022-03-29T17:50:25.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:25.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:50:25.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 40.497299194s 2022-03-29T17:50:25.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.497299194s (sent 48.502571106s) 2022-03-29T17:50:25.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:25.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,806 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,807 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:25.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,808 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:25.844 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,817 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:25.844 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,817 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:25.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,817 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:50:25.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,818 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 40.447814941s 2022-03-29T17:50:25.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,818 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.447814941s (sent 48.451026917s) 2022-03-29T17:50:25.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,818 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,818 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:25.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,818 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,818 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:25.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,818 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.447814941s ub 48.451026917s int 16.000000000s))): start 2022-03-29T17:50:25.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,819 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.447814941s ub 48.451026917s int 16.000000000s))): got map 13 2022-03-29T17:50:25.848 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,819 [shard 0] osd - peering_event(id=33, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:25.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,819 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.447814941s ub 48.451026917s int 16.000000000s))): pg present 2022-03-29T17:50:25.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,819 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.447814941s ub 48.451026917s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:25.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 40.447814941s ub 48.451026917s int 16.000000000s) 2022-03-29T17:50:25.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 48.515773773s 2022-03-29T17:50:25.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 48.515773773s 2022-03-29T17:50:25.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,819 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,820 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.447814941s ub 48.451026917s int 16.000000000s))): submitting ctx 2022-03-29T17:50:25.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,820 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,820 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.447814941s ub 48.451026917s int 16.000000000s))): complete 2022-03-29T17:50:25.854 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,827 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:25.854 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,827 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:25.854 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,827 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:50:25.854 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,827 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 40.457794189s 2022-03-29T17:50:25.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,827 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.457794189s (sent 48.460891724s) 2022-03-29T17:50:25.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:25.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:25.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451026917s))): start 2022-03-29T17:50:25.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451026917s))): got map 13 2022-03-29T17:50:25.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - peering_event(id=34, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:25.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451026917s))): pg present 2022-03-29T17:50:25.857 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451026917s) for pg: 2.0 2022-03-29T17:50:25.857 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 48.451026917s 2022-03-29T17:50:25.857 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,828 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.451026917s (sent 48.451026917s) 2022-03-29T17:50:25.857 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,829 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.857 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,829 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451026917s))): submitting ctx 2022-03-29T17:50:25.858 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,829 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,829 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:25.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,829 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.457794189s ub 48.460891724s int 16.000000000s))): start 2022-03-29T17:50:25.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,829 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.457794189s ub 48.460891724s int 16.000000000s))): got map 13 2022-03-29T17:50:25.859 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - peering_event(id=35, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451026917s))): complete 2022-03-29T17:50:25.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.457794189s ub 48.460891724s int 16.000000000s))): pg present 2022-03-29T17:50:25.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.457794189s ub 48.460891724s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:25.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 40.457794189s ub 48.460891724s int 16.000000000s) 2022-03-29T17:50:25.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 48.526576996s 2022-03-29T17:50:25.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 48.526576996s 2022-03-29T17:50:25.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.457794189s ub 48.460891724s int 16.000000000s))): submitting ctx 2022-03-29T17:50:25.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,830 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,831 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 40.457794189s ub 48.460891724s int 16.000000000s))): complete 2022-03-29T17:50:25.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,842 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:25.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,842 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:25.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,842 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:50:25.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,842 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 40.446838379s 2022-03-29T17:50:25.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,843 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 40.446838379s (sent 48.451824188s) 2022-03-29T17:50:25.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,843 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,843 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:25.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,843 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,843 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:25.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,843 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497524261s ub 48.502147675s int 16.000000000s))): start 2022-03-29T17:50:25.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,843 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497524261s ub 48.502147675s int 16.000000000s))): got map 13 2022-03-29T17:50:25.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,843 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:25.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497524261s ub 48.502147675s int 16.000000000s))): pg present 2022-03-29T17:50:25.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:25.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497299194s ub 48.502571106s int 16.000000000s))): start 2022-03-29T17:50:25.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497524261s ub 48.502147675s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:25.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 40.497524261s ub 48.502147675s int 16.000000000s) 2022-03-29T17:50:25.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 48.453380585s 2022-03-29T17:50:25.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 48.453380585s 2022-03-29T17:50:25.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497524261s ub 48.502147675s int 16.000000000s))): submitting ctx 2022-03-29T17:50:25.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497299194s ub 48.502571106s int 16.000000000s))): got map 13 2022-03-29T17:50:25.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,844 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497299194s ub 48.502571106s int 16.000000000s))): pg present 2022-03-29T17:50:25.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,845 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497299194s ub 48.502571106s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:25.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,845 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 40.497299194s ub 48.502571106s int 16.000000000s) 2022-03-29T17:50:25.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,845 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 48.454032898s 2022-03-29T17:50:25.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,845 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 48.454032898s 2022-03-29T17:50:25.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,845 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,845 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497299194s ub 48.502571106s int 16.000000000s))): submitting ctx 2022-03-29T17:50:25.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,845 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,845 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497524261s ub 48.502147675s int 16.000000000s))): complete 2022-03-29T17:50:25.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,846 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:25.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,846 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 40.446838379s ub 48.451824188s int 16.000000000s))): start 2022-03-29T17:50:25.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,846 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 40.497299194s ub 48.502571106s int 16.000000000s))): complete 2022-03-29T17:50:25.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,846 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 40.446838379s ub 48.451824188s int 16.000000000s))): got map 13 2022-03-29T17:50:25.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,847 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 40.446838379s ub 48.451824188s int 16.000000000s))): pg present 2022-03-29T17:50:25.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,847 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 40.446838379s ub 48.451824188s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:25.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,847 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 40.446838379s ub 48.451824188s int 16.000000000s) 2022-03-29T17:50:25.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,847 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 48.543449402s 2022-03-29T17:50:25.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,847 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 48.543449402s 2022-03-29T17:50:25.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,847 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,847 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 40.446838379s ub 48.451824188s int 16.000000000s))): submitting ctx 2022-03-29T17:50:25.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,847 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:25.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451824188s))): start 2022-03-29T17:50:25.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451824188s))): got map 13 2022-03-29T17:50:25.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451824188s))): pg present 2022-03-29T17:50:25.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451824188s) for pg: 2.2 2022-03-29T17:50:25.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 48.451824188s 2022-03-29T17:50:25.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.451824188s (sent 48.451824188s) 2022-03-29T17:50:25.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451824188s))): submitting ctx 2022-03-29T17:50:25.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,848 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,849 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 40.446838379s ub 48.451824188s int 16.000000000s))): complete 2022-03-29T17:50:25.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:25,849 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.451824188s))): complete 2022-03-29T17:50:25.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,849 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:25.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,849 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502147675s))): start 2022-03-29T17:50:25.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502147675s))): got map 13 2022-03-29T17:50:25.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502147675s))): pg present 2022-03-29T17:50:25.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502147675s) for pg: 2.6 2022-03-29T17:50:25.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 48.502147675s 2022-03-29T17:50:25.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.502147675s (sent 48.502147675s) 2022-03-29T17:50:25.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502147675s))): submitting ctx 2022-03-29T17:50:25.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,850 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:25.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502571106s))): start 2022-03-29T17:50:25.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502571106s))): got map 13 2022-03-29T17:50:25.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502571106s))): pg present 2022-03-29T17:50:25.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502147675s))): complete 2022-03-29T17:50:25.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502571106s) for pg: 2.7 2022-03-29T17:50:25.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 48.502571106s 2022-03-29T17:50:25.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.502571106s (sent 48.502571106s) 2022-03-29T17:50:25.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502571106s))): submitting ctx 2022-03-29T17:50:25.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:25,852 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 48.502571106s))): complete 2022-03-29T17:50:25.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,870 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:25.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,870 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.460891724s))): start 2022-03-29T17:50:25.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,871 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.460891724s))): got map 13 2022-03-29T17:50:25.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,871 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.460891724s))): pg present 2022-03-29T17:50:25.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,871 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.460891724s) for pg: 2.1 2022-03-29T17:50:25.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,871 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 48.460891724s 2022-03-29T17:50:25.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,871 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.460891724s (sent 48.460891724s) 2022-03-29T17:50:25.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,871 [shard 0] osd - log is not dirty 2022-03-29T17:50:25.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,871 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.460891724s))): submitting ctx 2022-03-29T17:50:25.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,871 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:25.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:25,872 [shard 0] osd - peering_event(id=36, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 48.460891724s))): complete 2022-03-29T17:50:26.347 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:26,345 [shard 0] alienstore - stat 2022-03-29T17:50:26.362 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:26,360 [shard 0] alienstore - stat 2022-03-29T17:50:26.362 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:26,361 [shard 0] alienstore - stat 2022-03-29T17:50:27.354 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:27,352 [shard 0] alienstore - stat 2022-03-29T17:50:27.368 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:27,367 [shard 0] alienstore - stat 2022-03-29T17:50:27.369 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:27,367 [shard 0] alienstore - stat 2022-03-29T17:50:28.360 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:28,359 [shard 0] alienstore - stat 2022-03-29T17:50:28.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:28,374 [shard 0] alienstore - stat 2022-03-29T17:50:28.376 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:28,374 [shard 0] alienstore - stat 2022-03-29T17:50:29.367 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:29,365 [shard 0] alienstore - stat 2022-03-29T17:50:29.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:29,380 [shard 0] alienstore - stat 2022-03-29T17:50:29.382 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:29,381 [shard 0] alienstore - stat 2022-03-29T17:50:30.374 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:30,371 [shard 0] alienstore - stat 2022-03-29T17:50:30.388 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:30,386 [shard 0] alienstore - stat 2022-03-29T17:50:30.389 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:30,387 [shard 0] alienstore - stat 2022-03-29T17:50:30.761 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,759 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:50:30.761 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,759 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:50:30.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,759 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:50:30.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,759 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 45.449344635s 2022-03-29T17:50:30.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,759 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 45.449344635s (sent 53.455188751s) 2022-03-29T17:50:30.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,759 [shard 0] osd - log is not dirty 2022-03-29T17:50:30.763 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,759 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:50:30.763 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,759 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:30.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,760 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:50:30.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,760 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 45.449344635s ub 53.455188751s int 16.000000000s))): start 2022-03-29T17:50:30.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,760 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 45.449344635s ub 53.455188751s int 16.000000000s))): got map 13 2022-03-29T17:50:30.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,760 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 45.449344635s ub 53.455188751s int 16.000000000s))): pg present 2022-03-29T17:50:30.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,760 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 45.449344635s ub 53.455188751s int 16.000000000s) for pg: 1.0 2022-03-29T17:50:30.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,760 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 45.449344635s ub 53.455188751s int 16.000000000s) 2022-03-29T17:50:30.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,760 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 53.369773865s 2022-03-29T17:50:30.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,760 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 53.369773865s 2022-03-29T17:50:30.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,761 [shard 0] osd - log is not dirty 2022-03-29T17:50:30.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,761 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 45.449344635s ub 53.455188751s int 16.000000000s))): submitting ctx 2022-03-29T17:50:30.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,761 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:30.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,761 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:50:30.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:30,762 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 45.449344635s ub 53.455188751s int 16.000000000s))): complete 2022-03-29T17:50:30.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,801 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:50:30.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,801 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 53.455188751s))): start 2022-03-29T17:50:30.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,801 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 53.455188751s))): got map 13 2022-03-29T17:50:30.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,802 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 53.455188751s))): pg present 2022-03-29T17:50:30.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,802 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 53.455188751s) for pg: 1.0 2022-03-29T17:50:30.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,802 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 53.455188751s 2022-03-29T17:50:30.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,802 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 53.455188751s (sent 53.455188751s) 2022-03-29T17:50:30.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,802 [shard 0] osd - log is not dirty 2022-03-29T17:50:30.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,802 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 53.455188751s))): submitting ctx 2022-03-29T17:50:30.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,802 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:30.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:30,803 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 53.455188751s))): complete 2022-03-29T17:50:31.379 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:31,377 [shard 0] alienstore - stat 2022-03-29T17:50:31.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:31,393 [shard 0] alienstore - stat 2022-03-29T17:50:31.395 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:31,393 [shard 0] alienstore - stat 2022-03-29T17:50:32.385 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:32,383 [shard 0] alienstore - stat 2022-03-29T17:50:32.401 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:32,399 [shard 0] alienstore - stat 2022-03-29T17:50:32.401 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:32,400 [shard 0] alienstore - stat 2022-03-29T17:50:33.391 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:33,389 [shard 0] alienstore - stat 2022-03-29T17:50:33.406 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:33,405 [shard 0] alienstore - stat 2022-03-29T17:50:33.407 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:33,405 [shard 0] alienstore - stat 2022-03-29T17:50:33.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,787 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:33.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,787 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:33.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,787 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:50:33.790 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,788 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 48.482231140s 2022-03-29T17:50:33.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,788 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.482231140s (sent 56.483932495s) 2022-03-29T17:50:33.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,788 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,788 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:33.791 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,788 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:33.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.482231140s ub 56.483932495s int 16.000000000s))): start 2022-03-29T17:50:33.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.482231140s ub 56.483932495s int 16.000000000s))): got map 13 2022-03-29T17:50:33.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.482231140s ub 56.483932495s int 16.000000000s))): pg present 2022-03-29T17:50:33.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.482231140s ub 56.483932495s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:33.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 48.482231140s ub 56.483932495s int 16.000000000s) 2022-03-29T17:50:33.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 56.422660828s 2022-03-29T17:50:33.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 56.422660828s 2022-03-29T17:50:33.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.482231140s ub 56.483932495s int 16.000000000s))): submitting ctx 2022-03-29T17:50:33.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,789 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,790 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:33.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,790 [shard 0] osd - peering_event(id=37, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.482231140s ub 56.483932495s int 16.000000000s))): complete 2022-03-29T17:50:33.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:33.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 56.483932495s))): start 2022-03-29T17:50:33.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 56.483932495s))): got map 13 2022-03-29T17:50:33.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 56.483932495s))): pg present 2022-03-29T17:50:33.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 56.483932495s) for pg: 2.3 2022-03-29T17:50:33.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 56.483932495s 2022-03-29T17:50:33.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.483932495s (sent 56.483932495s) 2022-03-29T17:50:33.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 56.483932495s))): submitting ctx 2022-03-29T17:50:33.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,791 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,793 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 56.483932495s))): complete 2022-03-29T17:50:33.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:33.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:33.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:33.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:33.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:50:33.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 48.492118835s 2022-03-29T17:50:33.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.492118835s (sent 56.494548798s) 2022-03-29T17:50:33.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:33.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,798 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:50:33.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,799 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 48.492549896s 2022-03-29T17:50:33.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,799 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.492549896s (sent 56.494888306s) 2022-03-29T17:50:33.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,799 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:33.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,799 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492118835s ub 56.494548798s int 16.000000000s))): start 2022-03-29T17:50:33.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,799 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492118835s ub 56.494548798s int 16.000000000s))): got map 13 2022-03-29T17:50:33.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,799 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492118835s ub 56.494548798s int 16.000000000s))): pg present 2022-03-29T17:50:33.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,799 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492118835s ub 56.494548798s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:33.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,799 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 48.492118835s ub 56.494548798s int 16.000000000s) 2022-03-29T17:50:33.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,800 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 56.408863068s 2022-03-29T17:50:33.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,800 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 56.408863068s 2022-03-29T17:50:33.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,800 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,800 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492118835s ub 56.494548798s int 16.000000000s))): submitting ctx 2022-03-29T17:50:33.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,800 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,800 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,800 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:33.810 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,800 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,801 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492118835s ub 56.494548798s int 16.000000000s))): complete 2022-03-29T17:50:33.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,801 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:33.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,801 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492549896s ub 56.494888306s int 16.000000000s))): start 2022-03-29T17:50:33.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,801 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492549896s ub 56.494888306s int 16.000000000s))): got map 13 2022-03-29T17:50:33.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,801 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492549896s ub 56.494888306s int 16.000000000s))): pg present 2022-03-29T17:50:33.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,801 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492549896s ub 56.494888306s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:33.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,801 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 48.492549896s ub 56.494888306s int 16.000000000s) 2022-03-29T17:50:33.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,801 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 56.410831451s 2022-03-29T17:50:33.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,802 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 56.410831451s 2022-03-29T17:50:33.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,802 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,802 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492549896s ub 56.494888306s int 16.000000000s))): submitting ctx 2022-03-29T17:50:33.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,802 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,802 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:33.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,802 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:33.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,802 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:33.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,802 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494548798s))): start 2022-03-29T17:50:33.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,803 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.492549896s ub 56.494888306s int 16.000000000s))): complete 2022-03-29T17:50:33.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,803 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494548798s))): got map 13 2022-03-29T17:50:33.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494548798s))): pg present 2022-03-29T17:50:33.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494548798s) for pg: 2.5 2022-03-29T17:50:33.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 56.494548798s 2022-03-29T17:50:33.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.494548798s (sent 56.494548798s) 2022-03-29T17:50:33.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494548798s))): submitting ctx 2022-03-29T17:50:33.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:33.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,804 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494888306s))): start 2022-03-29T17:50:33.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494548798s))): complete 2022-03-29T17:50:33.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494888306s))): got map 13 2022-03-29T17:50:33.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494888306s))): pg present 2022-03-29T17:50:33.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494888306s) for pg: 2.4 2022-03-29T17:50:33.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 56.494888306s 2022-03-29T17:50:33.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.494888306s (sent 56.494888306s) 2022-03-29T17:50:33.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494888306s))): submitting ctx 2022-03-29T17:50:33.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,805 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,806 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.494888306s))): complete 2022-03-29T17:50:33.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:33.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:33.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:33.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:33.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:50:33.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 48.502571106s 2022-03-29T17:50:33.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.502571106s (sent 56.504360199s) 2022-03-29T17:50:33.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:33.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:50:33.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 48.502147675s 2022-03-29T17:50:33.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.502147675s (sent 56.504695892s) 2022-03-29T17:50:33.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:33.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,808 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,809 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:33.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,809 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:33.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,817 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:33.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,817 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:33.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,817 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:50:33.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,817 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 48.451026917s 2022-03-29T17:50:33.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,818 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.451026917s (sent 56.450992584s) 2022-03-29T17:50:33.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,818 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,818 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:33.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,818 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,818 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:33.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,818 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.451026917s ub 56.450992584s int 16.000000000s))): start 2022-03-29T17:50:33.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,818 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.451026917s ub 56.450992584s int 16.000000000s))): got map 13 2022-03-29T17:50:33.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.451026917s ub 56.450992584s int 16.000000000s))): pg present 2022-03-29T17:50:33.834 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - peering_event(id=38, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:33.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.451026917s ub 56.450992584s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:33.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 48.451026917s ub 56.450992584s int 16.000000000s) 2022-03-29T17:50:33.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 56.515621185s 2022-03-29T17:50:33.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 56.515621185s 2022-03-29T17:50:33.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.451026917s ub 56.450992584s int 16.000000000s))): submitting ctx 2022-03-29T17:50:33.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,819 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,820 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.451026917s ub 56.450992584s int 16.000000000s))): complete 2022-03-29T17:50:33.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,827 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:33.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,827 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:33.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,827 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:50:33.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,827 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 48.460891724s 2022-03-29T17:50:33.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.460891724s (sent 56.461002350s) 2022-03-29T17:50:33.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:33.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:33.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.450992584s))): start 2022-03-29T17:50:33.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - peering_event(id=39, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:33.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.450992584s))): got map 13 2022-03-29T17:50:33.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.450992584s))): pg present 2022-03-29T17:50:33.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.450992584s) for pg: 2.0 2022-03-29T17:50:33.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,828 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 56.450992584s 2022-03-29T17:50:33.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,829 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.450992584s (sent 56.450992584s) 2022-03-29T17:50:33.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,829 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.844 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,829 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.450992584s))): submitting ctx 2022-03-29T17:50:33.844 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,829 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,829 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:33.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,829 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.460891724s ub 56.461002350s int 16.000000000s))): start 2022-03-29T17:50:33.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - peering_event(id=40, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.450992584s))): complete 2022-03-29T17:50:33.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,829 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.460891724s ub 56.461002350s int 16.000000000s))): got map 13 2022-03-29T17:50:33.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.460891724s ub 56.461002350s int 16.000000000s))): pg present 2022-03-29T17:50:33.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.460891724s ub 56.461002350s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:33.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 48.460891724s ub 56.461002350s int 16.000000000s) 2022-03-29T17:50:33.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 56.526622772s 2022-03-29T17:50:33.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 56.526622772s 2022-03-29T17:50:33.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.460891724s ub 56.461002350s int 16.000000000s))): submitting ctx 2022-03-29T17:50:33.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,830 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,831 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 48.460891724s ub 56.461002350s int 16.000000000s))): complete 2022-03-29T17:50:33.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,845 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:33.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,845 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502571106s ub 56.504360199s int 16.000000000s))): start 2022-03-29T17:50:33.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,845 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502571106s ub 56.504360199s int 16.000000000s))): got map 13 2022-03-29T17:50:33.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,845 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502571106s ub 56.504360199s int 16.000000000s))): pg present 2022-03-29T17:50:33.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:33.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502147675s ub 56.504695892s int 16.000000000s))): start 2022-03-29T17:50:33.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502571106s ub 56.504360199s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:33.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 48.502571106s ub 56.504360199s int 16.000000000s) 2022-03-29T17:50:33.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 56.455177307s 2022-03-29T17:50:33.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 56.455177307s 2022-03-29T17:50:33.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502571106s ub 56.504360199s int 16.000000000s))): submitting ctx 2022-03-29T17:50:33.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502147675s ub 56.504695892s int 16.000000000s))): got map 13 2022-03-29T17:50:33.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502147675s ub 56.504695892s int 16.000000000s))): pg present 2022-03-29T17:50:33.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502147675s ub 56.504695892s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:33.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 48.502147675s ub 56.504695892s int 16.000000000s) 2022-03-29T17:50:33.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 56.455818176s 2022-03-29T17:50:33.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,846 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 56.455818176s 2022-03-29T17:50:33.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,847 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,847 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502147675s ub 56.504695892s int 16.000000000s))): submitting ctx 2022-03-29T17:50:33.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,847 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,847 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502571106s ub 56.504360199s int 16.000000000s))): complete 2022-03-29T17:50:33.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,847 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 48.502147675s ub 56.504695892s int 16.000000000s))): complete 2022-03-29T17:50:33.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,851 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:33.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,852 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:33.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,852 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:50:33.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,852 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 48.451824188s 2022-03-29T17:50:33.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,852 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 48.451824188s (sent 56.461032867s) 2022-03-29T17:50:33.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,852 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,852 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:33.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,852 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,852 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:33.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:33.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461002350s))): start 2022-03-29T17:50:33.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461002350s))): got map 13 2022-03-29T17:50:33.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461002350s))): pg present 2022-03-29T17:50:33.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461002350s) for pg: 2.1 2022-03-29T17:50:33.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 56.461002350s 2022-03-29T17:50:33.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.461002350s (sent 56.461002350s) 2022-03-29T17:50:33.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,870 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461002350s))): submitting ctx 2022-03-29T17:50:33.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,871 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:33,871 [shard 0] osd - peering_event(id=41, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461002350s))): complete 2022-03-29T17:50:33.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,886 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:33.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,886 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504360199s))): start 2022-03-29T17:50:33.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,887 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504360199s))): got map 13 2022-03-29T17:50:33.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,887 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504360199s))): pg present 2022-03-29T17:50:33.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,887 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:33.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,887 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504695892s))): start 2022-03-29T17:50:33.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,887 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504360199s) for pg: 2.7 2022-03-29T17:50:33.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 56.504360199s 2022-03-29T17:50:33.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.504360199s (sent 56.504360199s) 2022-03-29T17:50:33.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504360199s))): submitting ctx 2022-03-29T17:50:33.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504695892s))): got map 13 2022-03-29T17:50:33.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504695892s))): pg present 2022-03-29T17:50:33.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504695892s) for pg: 2.6 2022-03-29T17:50:33.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 56.504695892s 2022-03-29T17:50:33.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.504695892s (sent 56.504695892s) 2022-03-29T17:50:33.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504695892s))): submitting ctx 2022-03-29T17:50:33.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,888 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:33.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 48.451824188s ub 56.461032867s int 16.000000000s))): start 2022-03-29T17:50:33.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504360199s))): complete 2022-03-29T17:50:33.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 48.451824188s ub 56.461032867s int 16.000000000s))): got map 13 2022-03-29T17:50:33.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 48.451824188s ub 56.461032867s int 16.000000000s))): pg present 2022-03-29T17:50:33.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 56.504695892s))): complete 2022-03-29T17:50:33.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 48.451824188s ub 56.461032867s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:33.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 48.451824188s ub 56.461032867s int 16.000000000s) 2022-03-29T17:50:33.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 56.585727692s 2022-03-29T17:50:33.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 56.585727692s 2022-03-29T17:50:33.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,889 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 48.451824188s ub 56.461032867s int 16.000000000s))): submitting ctx 2022-03-29T17:50:33.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,890 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:33,892 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 48.451824188s ub 56.461032867s int 16.000000000s))): complete 2022-03-29T17:50:33.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,927 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:33.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,927 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461032867s))): start 2022-03-29T17:50:33.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,928 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461032867s))): got map 13 2022-03-29T17:50:33.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,928 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461032867s))): pg present 2022-03-29T17:50:33.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,928 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461032867s) for pg: 2.2 2022-03-29T17:50:33.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,928 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 56.461032867s 2022-03-29T17:50:33.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,928 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.461032867s (sent 56.461032867s) 2022-03-29T17:50:33.932 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,928 [shard 0] osd - log is not dirty 2022-03-29T17:50:33.932 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,928 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461032867s))): submitting ctx 2022-03-29T17:50:33.932 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,928 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:33.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:33,929 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 56.461032867s))): complete 2022-03-29T17:50:34.397 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:34,395 [shard 0] alienstore - stat 2022-03-29T17:50:34.412 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:34,411 [shard 0] alienstore - stat 2022-03-29T17:50:34.413 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:34,412 [shard 0] alienstore - stat 2022-03-29T17:50:35.402 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:35,401 [shard 0] alienstore - stat 2022-03-29T17:50:35.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:35,417 [shard 0] alienstore - stat 2022-03-29T17:50:35.419 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:35,418 [shard 0] alienstore - stat 2022-03-29T17:50:36.408 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:36,406 [shard 0] alienstore - stat 2022-03-29T17:50:36.423 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:36,422 [shard 0] alienstore - stat 2022-03-29T17:50:36.424 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:36,422 [shard 0] alienstore - stat 2022-03-29T17:50:37.413 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:37,412 [shard 0] alienstore - stat 2022-03-29T17:50:37.429 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:37,428 [shard 0] alienstore - stat 2022-03-29T17:50:37.430 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:37,428 [shard 0] alienstore - stat 2022-03-29T17:50:38.420 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:38,418 [shard 0] alienstore - stat 2022-03-29T17:50:38.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:38,435 [shard 0] alienstore - stat 2022-03-29T17:50:38.437 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:38,436 [shard 0] alienstore - stat 2022-03-29T17:50:38.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,767 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:50:38.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,767 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:50:38.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,767 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:50:38.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,767 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 53.455188751s 2022-03-29T17:50:38.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,767 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 53.455188751s (sent 61.463680267s) 2022-03-29T17:50:38.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,768 [shard 0] osd - log is not dirty 2022-03-29T17:50:38.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,768 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:50:38.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,768 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:38.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,768 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:50:38.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,768 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 53.455188751s ub 61.463680267s int 16.000000000s))): start 2022-03-29T17:50:38.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 53.455188751s ub 61.463680267s int 16.000000000s))): got map 13 2022-03-29T17:50:38.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 53.455188751s ub 61.463680267s int 16.000000000s))): pg present 2022-03-29T17:50:38.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 53.455188751s ub 61.463680267s int 16.000000000s) for pg: 1.0 2022-03-29T17:50:38.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 53.455188751s ub 61.463680267s int 16.000000000s) 2022-03-29T17:50:38.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 61.378421783s 2022-03-29T17:50:38.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 61.378421783s 2022-03-29T17:50:38.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - log is not dirty 2022-03-29T17:50:38.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 53.455188751s ub 61.463680267s int 16.000000000s))): submitting ctx 2022-03-29T17:50:38.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,769 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:38.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:38,770 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 53.455188751s ub 61.463680267s int 16.000000000s))): complete 2022-03-29T17:50:38.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,770 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:50:38.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,771 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:50:38.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,771 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 61.463680267s))): start 2022-03-29T17:50:38.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,771 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 61.463680267s))): got map 13 2022-03-29T17:50:38.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,771 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 61.463680267s))): pg present 2022-03-29T17:50:38.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,771 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 61.463680267s) for pg: 1.0 2022-03-29T17:50:38.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,771 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 61.463680267s 2022-03-29T17:50:38.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,771 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 61.463680267s (sent 61.463680267s) 2022-03-29T17:50:38.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,772 [shard 0] osd - log is not dirty 2022-03-29T17:50:38.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,772 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 61.463680267s))): submitting ctx 2022-03-29T17:50:38.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,772 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:38.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:38,772 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 61.463680267s))): complete 2022-03-29T17:50:39.426 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:39,424 [shard 0] alienstore - stat 2022-03-29T17:50:39.443 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:39,441 [shard 0] alienstore - stat 2022-03-29T17:50:39.443 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:39,442 [shard 0] alienstore - stat 2022-03-29T17:50:40.432 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:40,431 [shard 0] alienstore - stat 2022-03-29T17:50:40.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:40,448 [shard 0] alienstore - stat 2022-03-29T17:50:40.451 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:40,449 [shard 0] alienstore - stat 2022-03-29T17:50:41.439 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:41,438 [shard 0] alienstore - stat 2022-03-29T17:50:41.456 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:41,455 [shard 0] alienstore - stat 2022-03-29T17:50:41.457 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:41,455 [shard 0] alienstore - stat 2022-03-29T17:50:41.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,797 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:41.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,797 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:41.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,797 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:50:41.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,797 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 56.483932495s 2022-03-29T17:50:41.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,797 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.483932495s (sent 64.493438721s) 2022-03-29T17:50:41.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,797 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,797 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:41.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,797 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,798 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:41.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,798 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.483932495s ub 64.493438721s int 16.000000000s))): start 2022-03-29T17:50:41.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,798 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.483932495s ub 64.493438721s int 16.000000000s))): got map 13 2022-03-29T17:50:41.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,798 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.483932495s ub 64.493438721s int 16.000000000s))): pg present 2022-03-29T17:50:41.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,799 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.483932495s ub 64.493438721s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:41.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,799 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 56.483932495s ub 64.493438721s int 16.000000000s) 2022-03-29T17:50:41.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,799 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 64.432205200s 2022-03-29T17:50:41.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,799 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 64.432205200s 2022-03-29T17:50:41.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,799 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,799 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.483932495s ub 64.493438721s int 16.000000000s))): submitting ctx 2022-03-29T17:50:41.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,799 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,800 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:41.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,800 [shard 0] osd - peering_event(id=42, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.483932495s ub 64.493438721s int 16.000000000s))): complete 2022-03-29T17:50:41.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,800 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:41.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,800 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 64.493438721s))): start 2022-03-29T17:50:41.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,800 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 64.493438721s))): got map 13 2022-03-29T17:50:41.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,801 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 64.493438721s))): pg present 2022-03-29T17:50:41.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,801 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 64.493438721s) for pg: 2.3 2022-03-29T17:50:41.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,801 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 64.493438721s 2022-03-29T17:50:41.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,801 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.493438721s (sent 64.493438721s) 2022-03-29T17:50:41.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,801 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,801 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 64.493438721s))): submitting ctx 2022-03-29T17:50:41.810 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,801 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.810 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,802 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 64.493438721s))): complete 2022-03-29T17:50:41.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:41.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:41.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:41.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:41.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:41.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:41.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:41.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:41.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:50:41.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 56.504695892s 2022-03-29T17:50:41.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.504695892s (sent 64.503684998s) 2022-03-29T17:50:41.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:41.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,807 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:50:41.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 56.494548798s 2022-03-29T17:50:41.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.494548798s (sent 64.503982544s) 2022-03-29T17:50:41.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:41.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:50:41.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 56.504360199s 2022-03-29T17:50:41.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.504360199s (sent 64.504188538s) 2022-03-29T17:50:41.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:41.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:50:41.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 56.494888306s 2022-03-29T17:50:41.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.494888306s (sent 64.504364014s) 2022-03-29T17:50:41.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:41.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,808 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,809 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:41.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,809 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504695892s ub 64.503684998s int 16.000000000s))): start 2022-03-29T17:50:41.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,809 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504695892s ub 64.503684998s int 16.000000000s))): got map 13 2022-03-29T17:50:41.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504695892s ub 64.503684998s int 16.000000000s))): pg present 2022-03-29T17:50:41.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:41.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494548798s ub 64.503982544s int 16.000000000s))): start 2022-03-29T17:50:41.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504695892s ub 64.503684998s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:41.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 56.504695892s ub 64.503684998s int 16.000000000s) 2022-03-29T17:50:41.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 64.419433594s 2022-03-29T17:50:41.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 64.419433594s 2022-03-29T17:50:41.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504695892s ub 64.503684998s int 16.000000000s))): submitting ctx 2022-03-29T17:50:41.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,810 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494548798s ub 64.503982544s int 16.000000000s))): got map 13 2022-03-29T17:50:41.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494548798s ub 64.503982544s int 16.000000000s))): pg present 2022-03-29T17:50:41.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:41.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504360199s ub 64.504188538s int 16.000000000s))): start 2022-03-29T17:50:41.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494548798s ub 64.503982544s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:41.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 56.494548798s ub 64.503982544s int 16.000000000s) 2022-03-29T17:50:41.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 64.420425415s 2022-03-29T17:50:41.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 64.420425415s 2022-03-29T17:50:41.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494548798s ub 64.503982544s int 16.000000000s))): submitting ctx 2022-03-29T17:50:41.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,811 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504695892s ub 64.503684998s int 16.000000000s))): complete 2022-03-29T17:50:41.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504360199s ub 64.504188538s int 16.000000000s))): got map 13 2022-03-29T17:50:41.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504360199s ub 64.504188538s int 16.000000000s))): pg present 2022-03-29T17:50:41.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494548798s ub 64.503982544s int 16.000000000s))): complete 2022-03-29T17:50:41.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504360199s ub 64.504188538s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:41.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 56.504360199s ub 64.504188538s int 16.000000000s) 2022-03-29T17:50:41.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 64.421424866s 2022-03-29T17:50:41.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 64.421424866s 2022-03-29T17:50:41.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504360199s ub 64.504188538s int 16.000000000s))): submitting ctx 2022-03-29T17:50:41.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,812 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:41.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494888306s ub 64.504364014s int 16.000000000s))): start 2022-03-29T17:50:41.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494888306s ub 64.504364014s int 16.000000000s))): got map 13 2022-03-29T17:50:41.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.504360199s ub 64.504188538s int 16.000000000s))): complete 2022-03-29T17:50:41.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494888306s ub 64.504364014s int 16.000000000s))): pg present 2022-03-29T17:50:41.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494888306s ub 64.504364014s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:41.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 56.494888306s ub 64.504364014s int 16.000000000s) 2022-03-29T17:50:41.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 64.422523499s 2022-03-29T17:50:41.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 64.422523499s 2022-03-29T17:50:41.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,813 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:41.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,814 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:41.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,814 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:41.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,814 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:41.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,814 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,814 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494888306s ub 64.504364014s int 16.000000000s))): submitting ctx 2022-03-29T17:50:41.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,814 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,815 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 56.494888306s ub 64.504364014s int 16.000000000s))): complete 2022-03-29T17:50:41.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,817 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:41.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,817 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:41.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,817 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:50:41.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,817 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 56.450992584s 2022-03-29T17:50:41.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,817 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.450992584s (sent 64.450630188s) 2022-03-29T17:50:41.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,817 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,817 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:41.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,817 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,818 [shard 0] osd - peering_event(id=43, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:41.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,818 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:41.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,819 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.450992584s ub 64.450630188s int 16.000000000s))): start 2022-03-29T17:50:41.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,819 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.450992584s ub 64.450630188s int 16.000000000s))): got map 13 2022-03-29T17:50:41.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,819 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.450992584s ub 64.450630188s int 16.000000000s))): pg present 2022-03-29T17:50:41.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,819 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.450992584s ub 64.450630188s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:41.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 56.450992584s ub 64.450630188s int 16.000000000s) 2022-03-29T17:50:41.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 64.515785217s 2022-03-29T17:50:41.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,819 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 64.515785217s 2022-03-29T17:50:41.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,820 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,820 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.450992584s ub 64.450630188s int 16.000000000s))): submitting ctx 2022-03-29T17:50:41.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,820 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,820 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.450992584s ub 64.450630188s int 16.000000000s))): complete 2022-03-29T17:50:41.849 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,827 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:41.849 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,827 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:41.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,827 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:50:41.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,827 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 56.461002350s 2022-03-29T17:50:41.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,827 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.461002350s (sent 64.460800171s) 2022-03-29T17:50:41.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,827 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,827 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:41.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,827 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:41.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.450630188s))): start 2022-03-29T17:50:41.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.450630188s))): got map 13 2022-03-29T17:50:41.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - peering_event(id=44, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:41.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.450630188s))): pg present 2022-03-29T17:50:41.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.450630188s) for pg: 2.0 2022-03-29T17:50:41.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 64.450630188s 2022-03-29T17:50:41.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.450630188s (sent 64.450630188s) 2022-03-29T17:50:41.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.450630188s))): submitting ctx 2022-03-29T17:50:41.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,828 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,829 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:41.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,829 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.461002350s ub 64.460800171s int 16.000000000s))): start 2022-03-29T17:50:41.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,829 [shard 0] osd - peering_event(id=45, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.450630188s))): complete 2022-03-29T17:50:41.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,829 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.461002350s ub 64.460800171s int 16.000000000s))): got map 13 2022-03-29T17:50:41.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,830 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.461002350s ub 64.460800171s int 16.000000000s))): pg present 2022-03-29T17:50:41.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,830 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.461002350s ub 64.460800171s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:41.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 56.461002350s ub 64.460800171s int 16.000000000s) 2022-03-29T17:50:41.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 64.526527405s 2022-03-29T17:50:41.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 64.526527405s 2022-03-29T17:50:41.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,830 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,830 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.461002350s ub 64.460800171s int 16.000000000s))): submitting ctx 2022-03-29T17:50:41.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,830 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,831 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 56.461002350s ub 64.460800171s int 16.000000000s))): complete 2022-03-29T17:50:41.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,850 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:41.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,850 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503684998s))): start 2022-03-29T17:50:41.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503684998s))): got map 13 2022-03-29T17:50:41.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503684998s))): pg present 2022-03-29T17:50:41.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503684998s) for pg: 2.6 2022-03-29T17:50:41.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:41.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:41.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:50:41.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 56.461032867s 2022-03-29T17:50:41.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 56.461032867s (sent 64.460609436s) 2022-03-29T17:50:41.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:41.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,852 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 64.503684998s 2022-03-29T17:50:41.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,852 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.503684998s (sent 64.503684998s) 2022-03-29T17:50:41.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,852 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,852 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503684998s))): submitting ctx 2022-03-29T17:50:41.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,852 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,852 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:41.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,853 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:41.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,853 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503982544s))): start 2022-03-29T17:50:41.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,853 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503684998s))): complete 2022-03-29T17:50:41.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,853 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503982544s))): got map 13 2022-03-29T17:50:41.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,853 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503982544s))): pg present 2022-03-29T17:50:41.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:41.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504188538s))): start 2022-03-29T17:50:41.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504188538s))): got map 13 2022-03-29T17:50:41.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503982544s) for pg: 2.5 2022-03-29T17:50:41.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 64.503982544s 2022-03-29T17:50:41.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.503982544s (sent 64.503982544s) 2022-03-29T17:50:41.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503982544s))): submitting ctx 2022-03-29T17:50:41.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504188538s))): pg present 2022-03-29T17:50:41.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504188538s) for pg: 2.7 2022-03-29T17:50:41.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 64.504188538s 2022-03-29T17:50:41.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,854 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.504188538s (sent 64.504188538s) 2022-03-29T17:50:41.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504188538s))): submitting ctx 2022-03-29T17:50:41.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.503982544s))): complete 2022-03-29T17:50:41.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:41.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504364014s))): start 2022-03-29T17:50:41.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504364014s))): got map 13 2022-03-29T17:50:41.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504364014s))): pg present 2022-03-29T17:50:41.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,855 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504364014s) for pg: 2.4 2022-03-29T17:50:41.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 64.504364014s 2022-03-29T17:50:41.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.504364014s (sent 64.504364014s) 2022-03-29T17:50:41.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504364014s))): submitting ctx 2022-03-29T17:50:41.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504188538s))): complete 2022-03-29T17:50:41.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:41.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 56.461032867s ub 64.460609436s int 16.000000000s))): start 2022-03-29T17:50:41.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 56.461032867s ub 64.460609436s int 16.000000000s))): got map 13 2022-03-29T17:50:41.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 56.461032867s ub 64.460609436s int 16.000000000s))): pg present 2022-03-29T17:50:41.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,856 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 64.504364014s))): complete 2022-03-29T17:50:41.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,857 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 56.461032867s ub 64.460609436s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:41.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,857 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 56.461032867s ub 64.460609436s int 16.000000000s) 2022-03-29T17:50:41.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,857 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 64.552986145s 2022-03-29T17:50:41.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,857 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 64.552986145s 2022-03-29T17:50:41.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,857 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,857 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 56.461032867s ub 64.460609436s int 16.000000000s))): submitting ctx 2022-03-29T17:50:41.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,857 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:41,857 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 56.461032867s ub 64.460609436s int 16.000000000s))): complete 2022-03-29T17:50:41.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,870 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:41.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,870 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460800171s))): start 2022-03-29T17:50:41.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,870 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460800171s))): got map 13 2022-03-29T17:50:41.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,870 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460800171s))): pg present 2022-03-29T17:50:41.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,871 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460800171s) for pg: 2.1 2022-03-29T17:50:41.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,871 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 64.460800171s 2022-03-29T17:50:41.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,871 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.460800171s (sent 64.460800171s) 2022-03-29T17:50:41.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,871 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,871 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460800171s))): submitting ctx 2022-03-29T17:50:41.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,871 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:41,872 [shard 0] osd - peering_event(id=46, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460800171s))): complete 2022-03-29T17:50:41.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,894 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:41.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,894 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460609436s))): start 2022-03-29T17:50:41.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,895 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460609436s))): got map 13 2022-03-29T17:50:41.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,895 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460609436s))): pg present 2022-03-29T17:50:41.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,895 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460609436s) for pg: 2.2 2022-03-29T17:50:41.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,895 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 64.460609436s 2022-03-29T17:50:41.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,895 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.460609436s (sent 64.460609436s) 2022-03-29T17:50:41.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,895 [shard 0] osd - log is not dirty 2022-03-29T17:50:41.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,895 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460609436s))): submitting ctx 2022-03-29T17:50:41.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,895 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:41.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:41,896 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 64.460609436s))): complete 2022-03-29T17:50:42.447 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:42,445 [shard 0] alienstore - stat 2022-03-29T17:50:42.463 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:42,461 [shard 0] alienstore - stat 2022-03-29T17:50:42.464 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:42,463 [shard 0] alienstore - stat 2022-03-29T17:50:43.454 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:43,452 [shard 0] alienstore - stat 2022-03-29T17:50:43.469 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:43,467 [shard 0] alienstore - stat 2022-03-29T17:50:43.471 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:43,470 [shard 0] alienstore - stat 2022-03-29T17:50:44.461 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:44,459 [shard 0] alienstore - stat 2022-03-29T17:50:44.476 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:44,474 [shard 0] alienstore - stat 2022-03-29T17:50:44.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:44,477 [shard 0] alienstore - stat 2022-03-29T17:50:45.467 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:45,465 [shard 0] alienstore - stat 2022-03-29T17:50:45.483 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:45,481 [shard 0] alienstore - stat 2022-03-29T17:50:45.485 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:45,483 [shard 0] alienstore - stat 2022-03-29T17:50:46.474 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:46,472 [shard 0] alienstore - stat 2022-03-29T17:50:46.489 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:46,488 [shard 0] alienstore - stat 2022-03-29T17:50:46.492 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:46,490 [shard 0] alienstore - stat 2022-03-29T17:50:46.775 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,772 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:50:46.775 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,772 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:50:46.775 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,772 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:50:46.775 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,772 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 61.463680267s 2022-03-29T17:50:46.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,772 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 61.463680267s (sent 69.468528748s) 2022-03-29T17:50:46.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,773 [shard 0] osd - log is not dirty 2022-03-29T17:50:46.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,773 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:50:46.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,773 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:46.778 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:50:46.778 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 61.463680267s ub 69.468528748s int 16.000000000s))): start 2022-03-29T17:50:46.778 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 61.463680267s ub 69.468528748s int 16.000000000s))): got map 13 2022-03-29T17:50:46.779 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 61.463680267s ub 69.468528748s int 16.000000000s))): pg present 2022-03-29T17:50:46.779 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 61.463680267s ub 69.468528748s int 16.000000000s) for pg: 1.0 2022-03-29T17:50:46.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 61.463680267s ub 69.468528748s int 16.000000000s) 2022-03-29T17:50:46.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 69.383728027s 2022-03-29T17:50:46.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 69.383728027s 2022-03-29T17:50:46.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,774 [shard 0] osd - log is not dirty 2022-03-29T17:50:46.781 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,775 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 61.463680267s ub 69.468528748s int 16.000000000s))): submitting ctx 2022-03-29T17:50:46.781 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,775 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:46.781 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,775 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:50:46.782 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,775 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:50:46.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:46,775 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 61.463680267s ub 69.468528748s int 16.000000000s))): complete 2022-03-29T17:50:46.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,775 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 69.468528748s))): start 2022-03-29T17:50:46.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,776 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 69.468528748s))): got map 13 2022-03-29T17:50:46.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,776 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 69.468528748s))): pg present 2022-03-29T17:50:46.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,776 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 69.468528748s) for pg: 1.0 2022-03-29T17:50:46.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,776 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 69.468528748s 2022-03-29T17:50:46.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,777 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 69.468528748s (sent 69.468528748s) 2022-03-29T17:50:46.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,777 [shard 0] osd - log is not dirty 2022-03-29T17:50:46.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,777 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 69.468528748s))): submitting ctx 2022-03-29T17:50:46.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,777 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:46.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:46,777 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 69.468528748s))): complete 2022-03-29T17:50:47.480 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:47,478 [shard 0] alienstore - stat 2022-03-29T17:50:47.496 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:47,494 [shard 0] alienstore - stat 2022-03-29T17:50:47.498 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:47,497 [shard 0] alienstore - stat 2022-03-29T17:50:48.488 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:48,485 [shard 0] alienstore - stat 2022-03-29T17:50:48.502 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:48,501 [shard 0] alienstore - stat 2022-03-29T17:50:48.505 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:48,504 [shard 0] alienstore - stat 2022-03-29T17:50:49.494 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:49,492 [shard 0] alienstore - stat 2022-03-29T17:50:49.509 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:49,507 [shard 0] alienstore - stat 2022-03-29T17:50:49.511 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:49,510 [shard 0] alienstore - stat 2022-03-29T17:50:49.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,802 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:49.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,802 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:49.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,802 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:50:49.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,802 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 64.493438721s 2022-03-29T17:50:49.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,802 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.493438721s (sent 72.498527527s) 2022-03-29T17:50:49.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,802 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,802 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:49.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,802 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,803 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:49.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,803 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.493438721s ub 72.498527527s int 16.000000000s))): start 2022-03-29T17:50:49.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,803 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.493438721s ub 72.498527527s int 16.000000000s))): got map 13 2022-03-29T17:50:49.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,803 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.493438721s ub 72.498527527s int 16.000000000s))): pg present 2022-03-29T17:50:49.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,804 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.493438721s ub 72.498527527s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:49.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,804 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 64.493438721s ub 72.498527527s int 16.000000000s) 2022-03-29T17:50:49.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,804 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 72.437156677s 2022-03-29T17:50:49.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,804 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 72.437156677s 2022-03-29T17:50:49.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,804 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,804 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.493438721s ub 72.498527527s int 16.000000000s))): submitting ctx 2022-03-29T17:50:49.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,804 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.810 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,805 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:49.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,805 [shard 0] osd - peering_event(id=47, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.493438721s ub 72.498527527s int 16.000000000s))): complete 2022-03-29T17:50:49.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,805 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:49.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,805 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 72.498527527s))): start 2022-03-29T17:50:49.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,805 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 72.498527527s))): got map 13 2022-03-29T17:50:49.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,806 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 72.498527527s))): pg present 2022-03-29T17:50:49.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,806 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 72.498527527s) for pg: 2.3 2022-03-29T17:50:49.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,806 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 72.498527527s 2022-03-29T17:50:49.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,806 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.498527527s (sent 72.498527527s) 2022-03-29T17:50:49.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,806 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,806 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 72.498527527s))): submitting ctx 2022-03-29T17:50:49.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,806 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,807 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 72.498527527s))): complete 2022-03-29T17:50:49.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:49.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:49.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:49.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:49.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:49.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:49.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:49.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:49.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:50:49.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 64.503982544s 2022-03-29T17:50:49.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,812 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.503982544s (sent 72.508766174s) 2022-03-29T17:50:49.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:49.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:50:49.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 64.504188538s 2022-03-29T17:50:49.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.504188538s (sent 72.509147644s) 2022-03-29T17:50:49.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:49.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,813 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:49.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503982544s ub 72.508766174s int 16.000000000s))): start 2022-03-29T17:50:49.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503982544s ub 72.508766174s int 16.000000000s))): got map 13 2022-03-29T17:50:49.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503982544s ub 72.508766174s int 16.000000000s))): pg present 2022-03-29T17:50:49.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503982544s ub 72.508766174s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:49.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 64.503982544s ub 72.508766174s int 16.000000000s) 2022-03-29T17:50:49.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 72.423530579s 2022-03-29T17:50:49.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 72.423530579s 2022-03-29T17:50:49.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503982544s ub 72.508766174s int 16.000000000s))): submitting ctx 2022-03-29T17:50:49.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,814 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,815 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:50:49.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,815 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 64.504364014s 2022-03-29T17:50:49.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,815 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.504364014s (sent 72.511436462s) 2022-03-29T17:50:49.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,815 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,815 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:49.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,815 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,815 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503982544s ub 72.508766174s int 16.000000000s))): complete 2022-03-29T17:50:49.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:49.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504188538s ub 72.509147644s int 16.000000000s))): start 2022-03-29T17:50:49.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504188538s ub 72.509147644s int 16.000000000s))): got map 13 2022-03-29T17:50:49.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504188538s ub 72.509147644s int 16.000000000s))): pg present 2022-03-29T17:50:49.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504188538s ub 72.509147644s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:49.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 64.504188538s ub 72.509147644s int 16.000000000s) 2022-03-29T17:50:49.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 72.425468445s 2022-03-29T17:50:49.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 72.425468445s 2022-03-29T17:50:49.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504188538s ub 72.509147644s int 16.000000000s))): submitting ctx 2022-03-29T17:50:49.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,816 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,817 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:50:49.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,817 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 64.503684998s 2022-03-29T17:50:49.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,817 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.503684998s (sent 72.513442993s) 2022-03-29T17:50:49.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,817 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,817 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504188538s ub 72.509147644s int 16.000000000s))): complete 2022-03-29T17:50:49.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:49.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504364014s ub 72.511436462s int 16.000000000s))): start 2022-03-29T17:50:49.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504364014s ub 72.511436462s int 16.000000000s))): got map 13 2022-03-29T17:50:49.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504364014s ub 72.511436462s int 16.000000000s))): pg present 2022-03-29T17:50:49.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504364014s ub 72.511436462s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:49.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 64.504364014s ub 72.511436462s int 16.000000000s) 2022-03-29T17:50:49.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 72.427497864s 2022-03-29T17:50:49.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 72.427497864s 2022-03-29T17:50:49.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504364014s ub 72.511436462s int 16.000000000s))): submitting ctx 2022-03-29T17:50:49.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,818 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,817 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:49.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,819 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,819 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:49.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,819 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:49.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,819 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:49.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,819 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:49.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,819 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:49.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,820 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:50:49.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,820 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 64.450630188s 2022-03-29T17:50:49.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,820 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.450630188s (sent 72.453125000s) 2022-03-29T17:50:49.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,820 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,820 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:49.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,820 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,820 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.504364014s ub 72.511436462s int 16.000000000s))): complete 2022-03-29T17:50:49.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,821 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:49.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,821 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503684998s ub 72.513442993s int 16.000000000s))): start 2022-03-29T17:50:49.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,821 [shard 0] osd - peering_event(id=48, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:49.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,821 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503684998s ub 72.513442993s int 16.000000000s))): got map 13 2022-03-29T17:50:49.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,821 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503684998s ub 72.513442993s int 16.000000000s))): pg present 2022-03-29T17:50:49.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,821 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503684998s ub 72.513442993s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:49.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,821 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 64.503684998s ub 72.513442993s int 16.000000000s) 2022-03-29T17:50:49.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,821 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 72.430816650s 2022-03-29T17:50:49.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,822 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 72.430816650s 2022-03-29T17:50:49.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,822 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,822 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503684998s ub 72.513442993s int 16.000000000s))): submitting ctx 2022-03-29T17:50:49.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,822 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,822 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:49.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,822 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 64.503684998s ub 72.513442993s int 16.000000000s))): complete 2022-03-29T17:50:49.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,823 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:49.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,823 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.508766174s))): start 2022-03-29T17:50:49.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,823 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:49.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,823 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.450630188s ub 72.453125000s int 16.000000000s))): start 2022-03-29T17:50:49.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,824 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.508766174s))): got map 13 2022-03-29T17:50:49.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,824 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.450630188s ub 72.453125000s int 16.000000000s))): got map 13 2022-03-29T17:50:49.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,824 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.508766174s))): pg present 2022-03-29T17:50:49.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,824 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.450630188s ub 72.453125000s int 16.000000000s))): pg present 2022-03-29T17:50:49.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:49.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.509147644s))): start 2022-03-29T17:50:49.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.450630188s ub 72.453125000s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:49.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 64.450630188s ub 72.453125000s int 16.000000000s) 2022-03-29T17:50:49.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 72.521423340s 2022-03-29T17:50:49.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 72.521423340s 2022-03-29T17:50:49.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.450630188s ub 72.453125000s int 16.000000000s))): submitting ctx 2022-03-29T17:50:49.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.508766174s) for pg: 2.5 2022-03-29T17:50:49.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 72.508766174s 2022-03-29T17:50:49.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.508766174s (sent 72.508766174s) 2022-03-29T17:50:49.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.508766174s))): submitting ctx 2022-03-29T17:50:49.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,825 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,826 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.509147644s))): got map 13 2022-03-29T17:50:49.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,826 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.509147644s))): pg present 2022-03-29T17:50:49.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.450630188s ub 72.453125000s int 16.000000000s))): complete 2022-03-29T17:50:49.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.508766174s))): complete 2022-03-29T17:50:49.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.509147644s) for pg: 2.7 2022-03-29T17:50:49.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 72.509147644s 2022-03-29T17:50:49.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.509147644s (sent 72.509147644s) 2022-03-29T17:50:49.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.509147644s))): submitting ctx 2022-03-29T17:50:49.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:49.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.511436462s))): start 2022-03-29T17:50:49.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,827 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.511436462s))): got map 13 2022-03-29T17:50:49.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.511436462s))): pg present 2022-03-29T17:50:49.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.509147644s))): complete 2022-03-29T17:50:49.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.511436462s) for pg: 2.4 2022-03-29T17:50:49.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 72.511436462s 2022-03-29T17:50:49.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.511436462s (sent 72.511436462s) 2022-03-29T17:50:49.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.511436462s))): submitting ctx 2022-03-29T17:50:49.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:49.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,828 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.513442993s))): start 2022-03-29T17:50:49.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.513442993s))): got map 13 2022-03-29T17:50:49.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.513442993s))): pg present 2022-03-29T17:50:49.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.511436462s))): complete 2022-03-29T17:50:49.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.513442993s) for pg: 2.6 2022-03-29T17:50:49.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 72.513442993s 2022-03-29T17:50:49.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.513442993s (sent 72.513442993s) 2022-03-29T17:50:49.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.513442993s))): submitting ctx 2022-03-29T17:50:49.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,829 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:49.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:49.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:50:49.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 64.460800171s 2022-03-29T17:50:49.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.460800171s (sent 72.463203430s) 2022-03-29T17:50:49.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:49.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:49.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,830 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.453125000s))): start 2022-03-29T17:50:49.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.453125000s))): got map 13 2022-03-29T17:50:49.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - peering_event(id=49, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:49.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.453125000s))): pg present 2022-03-29T17:50:49.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.453125000s) for pg: 2.0 2022-03-29T17:50:49.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 72.453125000s 2022-03-29T17:50:49.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.453125000s (sent 72.453125000s) 2022-03-29T17:50:49.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.453125000s))): submitting ctx 2022-03-29T17:50:49.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,831 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 72.513442993s))): complete 2022-03-29T17:50:49.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,832 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:49.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,832 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.460800171s ub 72.463203430s int 16.000000000s))): start 2022-03-29T17:50:49.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,832 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.460800171s ub 72.463203430s int 16.000000000s))): got map 13 2022-03-29T17:50:49.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,832 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.460800171s ub 72.463203430s int 16.000000000s))): pg present 2022-03-29T17:50:49.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,833 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.460800171s ub 72.463203430s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:49.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,833 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 64.460800171s ub 72.463203430s int 16.000000000s) 2022-03-29T17:50:49.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,833 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 72.528984070s 2022-03-29T17:50:49.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,833 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 72.528984070s 2022-03-29T17:50:49.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,833 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,833 [shard 0] osd - peering_event(id=50, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.453125000s))): complete 2022-03-29T17:50:49.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,833 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.460800171s ub 72.463203430s int 16.000000000s))): submitting ctx 2022-03-29T17:50:49.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,833 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,834 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 64.460800171s ub 72.463203430s int 16.000000000s))): complete 2022-03-29T17:50:49.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,855 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:49.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,855 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:49.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,855 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:50:49.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,855 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 64.460609436s 2022-03-29T17:50:49.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,855 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 64.460609436s (sent 72.464584351s) 2022-03-29T17:50:49.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,855 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,855 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:49.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,855 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,856 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:49.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,856 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:49.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,856 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 64.460609436s ub 72.464584351s int 16.000000000s))): start 2022-03-29T17:50:49.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 64.460609436s ub 72.464584351s int 16.000000000s))): got map 13 2022-03-29T17:50:49.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 64.460609436s ub 72.464584351s int 16.000000000s))): pg present 2022-03-29T17:50:49.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 64.460609436s ub 72.464584351s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:49.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 64.460609436s ub 72.464584351s int 16.000000000s) 2022-03-29T17:50:49.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 72.553329468s 2022-03-29T17:50:49.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 72.553329468s 2022-03-29T17:50:49.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 64.460609436s ub 72.464584351s int 16.000000000s))): submitting ctx 2022-03-29T17:50:49.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,857 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:49,858 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 64.460609436s ub 72.464584351s int 16.000000000s))): complete 2022-03-29T17:50:49.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:49.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.463203430s))): start 2022-03-29T17:50:49.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.463203430s))): got map 13 2022-03-29T17:50:49.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.463203430s))): pg present 2022-03-29T17:50:49.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.463203430s) for pg: 2.1 2022-03-29T17:50:49.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 72.463203430s 2022-03-29T17:50:49.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.463203430s (sent 72.463203430s) 2022-03-29T17:50:49.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,872 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.463203430s))): submitting ctx 2022-03-29T17:50:49.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,873 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:49,873 [shard 0] osd - peering_event(id=51, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.463203430s))): complete 2022-03-29T17:50:49.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,896 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:49.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,896 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.464584351s))): start 2022-03-29T17:50:49.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,896 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.464584351s))): got map 13 2022-03-29T17:50:49.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,896 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.464584351s))): pg present 2022-03-29T17:50:49.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,897 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.464584351s) for pg: 2.2 2022-03-29T17:50:49.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,897 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 72.464584351s 2022-03-29T17:50:49.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,897 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.464584351s (sent 72.464584351s) 2022-03-29T17:50:49.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,897 [shard 0] osd - log is not dirty 2022-03-29T17:50:49.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,897 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.464584351s))): submitting ctx 2022-03-29T17:50:49.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,897 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:49.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:49,897 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 72.464584351s))): complete 2022-03-29T17:50:50.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:50,499 [shard 0] alienstore - stat 2022-03-29T17:50:50.516 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:50,514 [shard 0] alienstore - stat 2022-03-29T17:50:50.519 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:50,518 [shard 0] alienstore - stat 2022-03-29T17:50:51.507 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:51,506 [shard 0] alienstore - stat 2022-03-29T17:50:51.522 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:51,521 [shard 0] alienstore - stat 2022-03-29T17:50:51.525 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:51,524 [shard 0] alienstore - stat 2022-03-29T17:50:52.514 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:52,512 [shard 0] alienstore - stat 2022-03-29T17:50:52.529 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:52,527 [shard 0] alienstore - stat 2022-03-29T17:50:52.532 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:52,530 [shard 0] alienstore - stat 2022-03-29T17:50:53.521 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:53,520 [shard 0] alienstore - stat 2022-03-29T17:50:53.536 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:53,534 [shard 0] alienstore - stat 2022-03-29T17:50:53.538 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:53,537 [shard 0] alienstore - stat 2022-03-29T17:50:54.528 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:54,526 [shard 0] alienstore - stat 2022-03-29T17:50:54.542 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:54,541 [shard 0] alienstore - stat 2022-03-29T17:50:54.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:54,544 [shard 0] alienstore - stat 2022-03-29T17:50:54.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,775 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:50:54.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,775 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:50:54.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,775 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:50:54.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,775 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 69.468528748s 2022-03-29T17:50:54.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,775 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 69.468528748s (sent 77.471633911s) 2022-03-29T17:50:54.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,775 [shard 0] osd - log is not dirty 2022-03-29T17:50:54.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,776 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:50:54.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,776 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:54.781 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,776 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:50:54.781 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,776 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 69.468528748s ub 77.471633911s int 16.000000000s))): start 2022-03-29T17:50:54.781 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 69.468528748s ub 77.471633911s int 16.000000000s))): got map 13 2022-03-29T17:50:54.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 69.468528748s ub 77.471633911s int 16.000000000s))): pg present 2022-03-29T17:50:54.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 69.468528748s ub 77.471633911s int 16.000000000s) for pg: 1.0 2022-03-29T17:50:54.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 69.468528748s ub 77.471633911s int 16.000000000s) 2022-03-29T17:50:54.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 77.386245728s 2022-03-29T17:50:54.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 77.386245728s 2022-03-29T17:50:54.783 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - log is not dirty 2022-03-29T17:50:54.783 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 69.468528748s ub 77.471633911s int 16.000000000s))): submitting ctx 2022-03-29T17:50:54.783 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,777 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:54.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,778 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:50:54.784 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:54,778 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 69.468528748s ub 77.471633911s int 16.000000000s))): complete 2022-03-29T17:50:54.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,778 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:50:54.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 77.471633911s))): start 2022-03-29T17:50:54.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 77.471633911s))): got map 13 2022-03-29T17:50:54.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 77.471633911s))): pg present 2022-03-29T17:50:54.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 77.471633911s) for pg: 1.0 2022-03-29T17:50:54.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 77.471633911s 2022-03-29T17:50:54.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 77.471633911s (sent 77.471633911s) 2022-03-29T17:50:54.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - log is not dirty 2022-03-29T17:50:54.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 77.471633911s))): submitting ctx 2022-03-29T17:50:54.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,779 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:54.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:54,780 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 77.471633911s))): complete 2022-03-29T17:50:55.535 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:55,533 [shard 0] alienstore - stat 2022-03-29T17:50:55.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:55,547 [shard 0] alienstore - stat 2022-03-29T17:50:55.555 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:55,550 [shard 0] alienstore - stat 2022-03-29T17:50:56.542 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:56,540 [shard 0] alienstore - stat 2022-03-29T17:50:56.555 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:56,554 [shard 0] alienstore - stat 2022-03-29T17:50:56.557 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:56,556 [shard 0] alienstore - stat 2022-03-29T17:50:56.736 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:56,735 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> unknown.? -@36880] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:36880/0 2022-03-29T17:50:56.737 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:56,735 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> unknown.? -@41188] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:41188/0 2022-03-29T17:50:56.737 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:56,736 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.? -@41188] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:50:56.742 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:56,736 [shard 0] monc - added challenge on [osd.2(client) v2:172.21.15.138:6803/35579 >> client.? -@41188] 2022-03-29T17:50:56.743 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,737 [shard 0] osd - handle_authentication mgr.x has caps allow * 2022-03-29T17:50:56.743 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:56,738 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4127 172.21.15.138:0/3450850499@41188] established: gs=0, pgs=4, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:56.743 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,738 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): start 2022-03-29T17:50:56.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,738 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): in repeat 2022-03-29T17:50:56.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,738 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]) same_interval_since: 11 2022-03-29T17:50:56.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,738 [shard 0] osd - OpSequencer::start_op: op=1, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:50:56.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - do_recover_missing check for recovery, 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:50:56.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): got obc lock 2022-03-29T17:50:56.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - with_head_obc 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:50:56.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - with_head_obc: cache miss on 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:50:56.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - load_metadata: object 2:88c1567c:::rbd_trash_purge_schedule:head doesn't exist, returning empty metadata 2022-03-29T17:50:56.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - load_head_obc: loaded obs 2:88c1567c:::rbd_trash_purge_schedule:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:50:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - load_head_obc: returning obc 2:88c1567c:::rbd_trash_purge_schedule:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:50:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:50:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,739 [shard 0] osd - handling op omap-get-vals on object 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:50:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,740 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:50:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,740 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8 - object 2:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:50:56.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,740 [shard 0] osd - with_head_obc: released 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:50:56.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,740 [shard 0] osd - OpSequencer::finish_op_in_order: op=1, last_started=1, last_unblocked=1, last_completed=0 2022-03-29T17:50:56.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:56,740 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): destroying 2022-03-29T17:50:56.751 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:56,741 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> client.? -@36880] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:50:56.752 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:56,741 [shard 0] monc - added challenge on [osd.0(client) v2:172.21.15.138:6805/35577 >> client.? -@36880] 2022-03-29T17:50:56.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,742 [shard 0] osd - handle_authentication mgr.x has caps allow * 2022-03-29T17:50:56.752 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:56,742 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> client.4127 172.21.15.138:0/3450850499@36880] established: gs=0, pgs=3, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:50:56.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,743 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:1 2.2 2.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): start 2022-03-29T17:50:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,743 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:1 2.2 2.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): in repeat 2022-03-29T17:50:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,743 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:1 2.2 2.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]) same_interval_since: 11 2022-03-29T17:50:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,743 [shard 0] osd - OpSequencer::start_op: op=1, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:50:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,743 [shard 0] osd - do_recover_missing check for recovery, 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:50:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,743 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): got obc lock 2022-03-29T17:50:56.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,743 [shard 0] osd - with_head_obc 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:50:56.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,743 [shard 0] osd - with_head_obc: cache miss on 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:50:56.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,744 [shard 0] osd - load_metadata: object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head doesn't exist, returning empty metadata 2022-03-29T17:50:56.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,744 [shard 0] osd - load_head_obc: loaded obs 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:50:56.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,744 [shard 0] osd - load_head_obc: returning obc 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:50:56.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,744 [shard 0] osd - do_osd_ops_execute: object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head - handling op omap-get-vals 2022-03-29T17:50:56.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,744 [shard 0] osd - handling op omap-get-vals on object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:50:56.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,744 [shard 0] osd - do_osd_ops_execute: object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head all operations successful 2022-03-29T17:50:56.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,744 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8 - object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head sending reply 2022-03-29T17:50:56.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,744 [shard 0] osd - with_head_obc: released 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:50:56.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,745 [shard 0] osd - OpSequencer::finish_op_in_order: op=1, last_started=1, last_unblocked=1, last_completed=0 2022-03-29T17:50:56.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:56,745 [shard 0] osd - client_request(id=1, detail=m=[osd_op(client.4127.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): destroying 2022-03-29T17:50:57.548 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:57,545 [shard 0] alienstore - stat 2022-03-29T17:50:57.562 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:57,560 [shard 0] alienstore - stat 2022-03-29T17:50:57.562 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:57,561 [shard 0] alienstore - stat 2022-03-29T17:50:57.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,801 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:57.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,801 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:57.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,801 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:50:57.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,801 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 72.498527527s 2022-03-29T17:50:57.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,801 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.498527527s (sent 80.497756958s) 2022-03-29T17:50:57.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,802 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,802 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:57.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,802 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,802 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:50:57.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,802 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.498527527s ub 80.497756958s int 16.000000000s))): start 2022-03-29T17:50:57.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.498527527s ub 80.497756958s int 16.000000000s))): got map 13 2022-03-29T17:50:57.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.498527527s ub 80.497756958s int 16.000000000s))): pg present 2022-03-29T17:50:57.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.498527527s ub 80.497756958s int 16.000000000s) for pg: 2.3 2022-03-29T17:50:57.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 72.498527527s ub 80.497756958s int 16.000000000s) 2022-03-29T17:50:57.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 80.436508179s 2022-03-29T17:50:57.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 80.436508179s 2022-03-29T17:50:57.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.498527527s ub 80.497756958s int 16.000000000s))): submitting ctx 2022-03-29T17:50:57.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,803 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.810 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,804 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:57.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,804 [shard 0] osd - peering_event(id=52, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.498527527s ub 80.497756958s int 16.000000000s))): complete 2022-03-29T17:50:57.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,804 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:50:57.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,804 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 80.497756958s))): start 2022-03-29T17:50:57.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,805 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 80.497756958s))): got map 13 2022-03-29T17:50:57.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,805 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 80.497756958s))): pg present 2022-03-29T17:50:57.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,805 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 80.497756958s) for pg: 2.3 2022-03-29T17:50:57.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,805 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 80.497756958s 2022-03-29T17:50:57.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,805 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.497756958s (sent 80.497756958s) 2022-03-29T17:50:57.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,805 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,805 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 80.497756958s))): submitting ctx 2022-03-29T17:50:57.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,805 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,806 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 80.497756958s))): complete 2022-03-29T17:50:57.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,821 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:57.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,821 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:57.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,821 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:50:57.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,821 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 72.453125000s 2022-03-29T17:50:57.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,821 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.453125000s (sent 80.454757690s) 2022-03-29T17:50:57.825 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,821 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.825 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,821 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:57.825 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,821 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:57.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:57.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:57.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:57.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:57.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:57.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:57.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:57.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:50:57.829 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,822 [shard 0] osd - peering_event(id=53, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:57.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 72.511436462s 2022-03-29T17:50:57.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.511436462s (sent 80.519104004s) 2022-03-29T17:50:57.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:57.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:50:57.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 72.508766174s 2022-03-29T17:50:57.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.508766174s (sent 80.519508362s) 2022-03-29T17:50:57.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:57.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,823 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:50:57.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 72.509147644s 2022-03-29T17:50:57.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.509147644s (sent 80.519996643s) 2022-03-29T17:50:57.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:57.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:50:57.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.511436462s ub 80.519104004s int 16.000000000s))): start 2022-03-29T17:50:57.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.511436462s ub 80.519104004s int 16.000000000s))): got map 13 2022-03-29T17:50:57.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,824 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.511436462s ub 80.519104004s int 16.000000000s))): pg present 2022-03-29T17:50:57.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.511436462s ub 80.519104004s int 16.000000000s) for pg: 2.4 2022-03-29T17:50:57.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 72.511436462s ub 80.519104004s int 16.000000000s) 2022-03-29T17:50:57.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 80.433959961s 2022-03-29T17:50:57.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 80.433959961s 2022-03-29T17:50:57.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.511436462s ub 80.519104004s int 16.000000000s))): submitting ctx 2022-03-29T17:50:57.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:50:57.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 72.513442993s 2022-03-29T17:50:57.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.513442993s (sent 80.521705627s) 2022-03-29T17:50:57.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:57.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,825 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,826 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.511436462s ub 80.519104004s int 16.000000000s))): complete 2022-03-29T17:50:57.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,826 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:57.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,826 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:57.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,827 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:50:57.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,827 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.453125000s ub 80.454757690s int 16.000000000s))): start 2022-03-29T17:50:57.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,827 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:57.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,827 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:50:57.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,827 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.508766174s ub 80.519508362s int 16.000000000s))): start 2022-03-29T17:50:57.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,827 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.508766174s ub 80.519508362s int 16.000000000s))): got map 13 2022-03-29T17:50:57.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,827 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.508766174s ub 80.519508362s int 16.000000000s))): pg present 2022-03-29T17:50:57.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:50:57.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.509147644s ub 80.519996643s int 16.000000000s))): start 2022-03-29T17:50:57.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.509147644s ub 80.519996643s int 16.000000000s))): got map 13 2022-03-29T17:50:57.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.508766174s ub 80.519508362s int 16.000000000s) for pg: 2.5 2022-03-29T17:50:57.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 72.508766174s ub 80.519508362s int 16.000000000s) 2022-03-29T17:50:57.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 80.437377930s 2022-03-29T17:50:57.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 80.437377930s 2022-03-29T17:50:57.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.508766174s ub 80.519508362s int 16.000000000s))): submitting ctx 2022-03-29T17:50:57.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,828 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.509147644s ub 80.519996643s int 16.000000000s))): pg present 2022-03-29T17:50:57.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:50:57.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.513442993s ub 80.521705627s int 16.000000000s))): start 2022-03-29T17:50:57.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.509147644s ub 80.519996643s int 16.000000000s) for pg: 2.7 2022-03-29T17:50:57.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 72.509147644s ub 80.519996643s int 16.000000000s) 2022-03-29T17:50:57.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 80.438217163s 2022-03-29T17:50:57.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 80.438217163s 2022-03-29T17:50:57.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.509147644s ub 80.519996643s int 16.000000000s))): submitting ctx 2022-03-29T17:50:57.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.508766174s ub 80.519508362s int 16.000000000s))): complete 2022-03-29T17:50:57.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.513442993s ub 80.521705627s int 16.000000000s))): got map 13 2022-03-29T17:50:57.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,829 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.513442993s ub 80.521705627s int 16.000000000s))): pg present 2022-03-29T17:50:57.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,830 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.513442993s ub 80.521705627s int 16.000000000s) for pg: 2.6 2022-03-29T17:50:57.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,830 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 72.513442993s ub 80.521705627s int 16.000000000s) 2022-03-29T17:50:57.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,830 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 80.438949585s 2022-03-29T17:50:57.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,830 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 80.438949585s 2022-03-29T17:50:57.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,830 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,830 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.513442993s ub 80.521705627s int 16.000000000s))): submitting ctx 2022-03-29T17:50:57.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,830 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,830 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.509147644s ub 80.519996643s int 16.000000000s))): complete 2022-03-29T17:50:57.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,831 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 72.513442993s ub 80.521705627s int 16.000000000s))): complete 2022-03-29T17:50:57.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,831 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:57.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,831 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:57.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,831 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:50:57.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,831 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 72.463203430s 2022-03-29T17:50:57.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,831 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.463203430s (sent 80.464958191s) 2022-03-29T17:50:57.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,832 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,832 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:57.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,832 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,832 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:57.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,832 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.453125000s ub 80.454757690s int 16.000000000s))): got map 13 2022-03-29T17:50:57.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,832 [shard 0] osd - peering_event(id=54, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:57.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,833 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.453125000s ub 80.454757690s int 16.000000000s))): pg present 2022-03-29T17:50:57.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,833 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:50:57.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,833 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.463203430s ub 80.464958191s int 16.000000000s))): start 2022-03-29T17:50:57.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,833 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.453125000s ub 80.454757690s int 16.000000000s) for pg: 2.0 2022-03-29T17:50:57.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,833 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 72.453125000s ub 80.454757690s int 16.000000000s) 2022-03-29T17:50:57.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,833 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 80.529708862s 2022-03-29T17:50:57.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,834 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 80.529708862s 2022-03-29T17:50:57.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,834 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,834 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.453125000s ub 80.454757690s int 16.000000000s))): submitting ctx 2022-03-29T17:50:57.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,834 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,834 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:50:57.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,834 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519104004s))): start 2022-03-29T17:50:57.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,834 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.463203430s ub 80.464958191s int 16.000000000s))): got map 13 2022-03-29T17:50:57.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.463203430s ub 80.464958191s int 16.000000000s))): pg present 2022-03-29T17:50:57.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519104004s))): got map 13 2022-03-29T17:50:57.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.453125000s ub 80.454757690s int 16.000000000s))): complete 2022-03-29T17:50:57.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.463203430s ub 80.464958191s int 16.000000000s) for pg: 2.1 2022-03-29T17:50:57.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 72.463203430s ub 80.464958191s int 16.000000000s) 2022-03-29T17:50:57.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 80.531494141s 2022-03-29T17:50:57.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 80.531494141s 2022-03-29T17:50:57.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.463203430s ub 80.464958191s int 16.000000000s))): submitting ctx 2022-03-29T17:50:57.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,835 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519104004s))): pg present 2022-03-29T17:50:57.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519104004s) for pg: 2.4 2022-03-29T17:50:57.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 80.519104004s 2022-03-29T17:50:57.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.519104004s (sent 80.519104004s) 2022-03-29T17:50:57.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519104004s))): submitting ctx 2022-03-29T17:50:57.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 72.463203430s ub 80.464958191s int 16.000000000s))): complete 2022-03-29T17:50:57.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:50:57.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,836 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519508362s))): start 2022-03-29T17:50:57.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519104004s))): complete 2022-03-29T17:50:57.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519508362s))): got map 13 2022-03-29T17:50:57.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519508362s))): pg present 2022-03-29T17:50:57.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519508362s) for pg: 2.5 2022-03-29T17:50:57.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 80.519508362s 2022-03-29T17:50:57.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.519508362s (sent 80.519508362s) 2022-03-29T17:50:57.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519508362s))): submitting ctx 2022-03-29T17:50:57.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,837 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:50:57.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519996643s))): start 2022-03-29T17:50:57.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519996643s))): got map 13 2022-03-29T17:50:57.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519508362s))): complete 2022-03-29T17:50:57.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519996643s))): pg present 2022-03-29T17:50:57.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519996643s) for pg: 2.7 2022-03-29T17:50:57.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 80.519996643s 2022-03-29T17:50:57.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.519996643s (sent 80.519996643s) 2022-03-29T17:50:57.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519996643s))): submitting ctx 2022-03-29T17:50:57.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:50:57.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.521705627s))): start 2022-03-29T17:50:57.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.521705627s))): got map 13 2022-03-29T17:50:57.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.521705627s))): pg present 2022-03-29T17:50:57.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.519996643s))): complete 2022-03-29T17:50:57.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.521705627s) for pg: 2.6 2022-03-29T17:50:57.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 80.521705627s 2022-03-29T17:50:57.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.521705627s (sent 80.521705627s) 2022-03-29T17:50:57.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.521705627s))): submitting ctx 2022-03-29T17:50:57.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,839 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,840 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 80.521705627s))): complete 2022-03-29T17:50:57.892 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,857 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:50:57.893 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,857 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:50:57.893 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,857 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:50:57.893 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,857 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 72.464584351s 2022-03-29T17:50:57.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,857 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 72.464584351s (sent 80.466499329s) 2022-03-29T17:50:57.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,857 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,857 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:50:57.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,857 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,858 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:50:57.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,858 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:50:57.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,858 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 72.464584351s ub 80.466499329s int 16.000000000s))): start 2022-03-29T17:50:57.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,858 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 72.464584351s ub 80.466499329s int 16.000000000s))): got map 13 2022-03-29T17:50:57.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 72.464584351s ub 80.466499329s int 16.000000000s))): pg present 2022-03-29T17:50:57.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 72.464584351s ub 80.466499329s int 16.000000000s) for pg: 2.2 2022-03-29T17:50:57.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 72.464584351s ub 80.466499329s int 16.000000000s) 2022-03-29T17:50:57.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 80.555122375s 2022-03-29T17:50:57.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 80.555122375s 2022-03-29T17:50:57.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 72.464584351s ub 80.466499329s int 16.000000000s))): submitting ctx 2022-03-29T17:50:57.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:50:57,859 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 72.464584351s ub 80.466499329s int 16.000000000s))): complete 2022-03-29T17:50:57.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,874 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:50:57.903 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,874 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.454757690s))): start 2022-03-29T17:50:57.903 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,874 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.454757690s))): got map 13 2022-03-29T17:50:57.903 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,874 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.454757690s))): pg present 2022-03-29T17:50:57.903 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:50:57.904 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.464958191s))): start 2022-03-29T17:50:57.904 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.454757690s) for pg: 2.0 2022-03-29T17:50:57.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 80.454757690s 2022-03-29T17:50:57.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.454757690s (sent 80.454757690s) 2022-03-29T17:50:57.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.454757690s))): submitting ctx 2022-03-29T17:50:57.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.464958191s))): got map 13 2022-03-29T17:50:57.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.464958191s))): pg present 2022-03-29T17:50:57.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.464958191s) for pg: 2.1 2022-03-29T17:50:57.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 80.464958191s 2022-03-29T17:50:57.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.464958191s (sent 80.464958191s) 2022-03-29T17:50:57.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.464958191s))): submitting ctx 2022-03-29T17:50:57.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,875 [shard 0] osd - peering_event(id=55, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.454757690s))): complete 2022-03-29T17:50:57.909 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:50:57,876 [shard 0] osd - peering_event(id=56, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.464958191s))): complete 2022-03-29T17:50:57.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,898 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:50:57.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,898 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.466499329s))): start 2022-03-29T17:50:57.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,898 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.466499329s))): got map 13 2022-03-29T17:50:57.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,898 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.466499329s))): pg present 2022-03-29T17:50:57.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,898 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.466499329s) for pg: 2.2 2022-03-29T17:50:57.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,899 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 80.466499329s 2022-03-29T17:50:57.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,899 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.466499329s (sent 80.466499329s) 2022-03-29T17:50:57.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,899 [shard 0] osd - log is not dirty 2022-03-29T17:50:57.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,899 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.466499329s))): submitting ctx 2022-03-29T17:50:57.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,899 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:50:57.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:50:57,899 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 80.466499329s))): complete 2022-03-29T17:50:58.551 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:58,550 [shard 0] alienstore - stat 2022-03-29T17:50:58.566 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:58,565 [shard 0] alienstore - stat 2022-03-29T17:50:58.567 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:58,565 [shard 0] alienstore - stat 2022-03-29T17:50:59.555 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:50:59,553 [shard 0] alienstore - stat 2022-03-29T17:50:59.570 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:50:59,568 [shard 0] alienstore - stat 2022-03-29T17:50:59.571 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:50:59,569 [shard 0] alienstore - stat 2022-03-29T17:51:00.560 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:00,558 [shard 0] alienstore - stat 2022-03-29T17:51:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:00,574 [shard 0] alienstore - stat 2022-03-29T17:51:00.576 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:00,574 [shard 0] alienstore - stat 2022-03-29T17:51:01.564 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:01,563 [shard 0] alienstore - stat 2022-03-29T17:51:01.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:01,579 [shard 0] alienstore - stat 2022-03-29T17:51:01.581 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:01,580 [shard 0] alienstore - stat 2022-03-29T17:51:02.569 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:02,567 [shard 0] alienstore - stat 2022-03-29T17:51:02.585 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:02,584 [shard 0] alienstore - stat 2022-03-29T17:51:02.586 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:02,584 [shard 0] alienstore - stat 2022-03-29T17:51:02.787 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,785 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:51:02.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,785 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:51:02.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,785 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:51:02.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,785 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 77.471633911s 2022-03-29T17:51:02.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,785 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 77.471633911s (sent 85.481430054s) 2022-03-29T17:51:02.788 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,785 [shard 0] osd - log is not dirty 2022-03-29T17:51:02.789 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,785 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:51:02.789 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,785 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:02.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,786 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:51:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,786 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 77.471633911s ub 85.481430054s int 16.000000000s))): start 2022-03-29T17:51:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,786 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 77.471633911s ub 85.481430054s int 16.000000000s))): got map 13 2022-03-29T17:51:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,786 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 77.471633911s ub 85.481430054s int 16.000000000s))): pg present 2022-03-29T17:51:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,786 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 77.471633911s ub 85.481430054s int 16.000000000s) for pg: 1.0 2022-03-29T17:51:02.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,786 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 77.471633911s ub 85.481430054s int 16.000000000s) 2022-03-29T17:51:02.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,786 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 85.395805359s 2022-03-29T17:51:02.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,786 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 85.395805359s 2022-03-29T17:51:02.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,787 [shard 0] osd - log is not dirty 2022-03-29T17:51:02.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,787 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 77.471633911s ub 85.481430054s int 16.000000000s))): submitting ctx 2022-03-29T17:51:02.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,787 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:02.795 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,787 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:51:02.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:02,787 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 77.471633911s ub 85.481430054s int 16.000000000s))): complete 2022-03-29T17:51:02.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:51:02.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 85.481430054s))): start 2022-03-29T17:51:02.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 85.481430054s))): got map 13 2022-03-29T17:51:02.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 85.481430054s))): pg present 2022-03-29T17:51:02.799 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 85.481430054s) for pg: 1.0 2022-03-29T17:51:02.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 85.481430054s 2022-03-29T17:51:02.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 85.481430054s (sent 85.481430054s) 2022-03-29T17:51:02.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - log is not dirty 2022-03-29T17:51:02.800 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 85.481430054s))): submitting ctx 2022-03-29T17:51:02.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,788 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:02.801 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:02,789 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 85.481430054s))): complete 2022-03-29T17:51:03.573 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:03,572 [shard 0] alienstore - stat 2022-03-29T17:51:03.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:03,588 [shard 0] alienstore - stat 2022-03-29T17:51:03.591 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:03,589 [shard 0] alienstore - stat 2022-03-29T17:51:04.578 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:04,576 [shard 0] alienstore - stat 2022-03-29T17:51:04.595 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:04,593 [shard 0] alienstore - stat 2022-03-29T17:51:04.595 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:04,594 [shard 0] alienstore - stat 2022-03-29T17:51:05.583 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:05,581 [shard 0] alienstore - stat 2022-03-29T17:51:05.599 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:05,598 [shard 0] alienstore - stat 2022-03-29T17:51:05.600 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:05,598 [shard 0] alienstore - stat 2022-03-29T17:51:05.811 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,809 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:05.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,809 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:05.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,809 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:51:05.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,809 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 80.497756958s 2022-03-29T17:51:05.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,809 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.497756958s (sent 88.505439758s) 2022-03-29T17:51:05.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,809 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,809 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:05.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,809 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:51:05.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.497756958s ub 88.505439758s int 16.000000000s))): start 2022-03-29T17:51:05.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.497756958s ub 88.505439758s int 16.000000000s))): got map 13 2022-03-29T17:51:05.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.497756958s ub 88.505439758s int 16.000000000s))): pg present 2022-03-29T17:51:05.816 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.497756958s ub 88.505439758s int 16.000000000s) for pg: 2.3 2022-03-29T17:51:05.816 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 80.497756958s ub 88.505439758s int 16.000000000s) 2022-03-29T17:51:05.816 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 88.443870544s 2022-03-29T17:51:05.816 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 88.443870544s 2022-03-29T17:51:05.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.497756958s ub 88.505439758s int 16.000000000s))): submitting ctx 2022-03-29T17:51:05.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,810 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,811 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:05.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,811 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:51:05.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,811 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 88.505439758s))): start 2022-03-29T17:51:05.819 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,812 [shard 0] osd - peering_event(id=57, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.497756958s ub 88.505439758s int 16.000000000s))): complete 2022-03-29T17:51:05.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,811 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 88.505439758s))): got map 13 2022-03-29T17:51:05.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,812 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 88.505439758s))): pg present 2022-03-29T17:51:05.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,812 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 88.505439758s) for pg: 2.3 2022-03-29T17:51:05.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,812 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 88.505439758s 2022-03-29T17:51:05.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,812 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.505439758s (sent 88.505439758s) 2022-03-29T17:51:05.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,812 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,812 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 88.505439758s))): submitting ctx 2022-03-29T17:51:05.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,812 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,813 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 88.505439758s))): complete 2022-03-29T17:51:05.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:05.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:05.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:51:05.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 80.454757690s 2022-03-29T17:51:05.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.454757690s (sent 88.462387085s) 2022-03-29T17:51:05.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:05.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:05.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:05.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:05.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:05.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:05.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,829 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:05.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:05.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:05.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:51:05.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 80.519508362s 2022-03-29T17:51:05.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.519508362s (sent 88.526153564s) 2022-03-29T17:51:05.837 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - peering_event(id=58, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:05.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:05.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,830 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:51:05.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519508362s ub 88.526153564s int 16.000000000s))): start 2022-03-29T17:51:05.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519508362s ub 88.526153564s int 16.000000000s))): got map 13 2022-03-29T17:51:05.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519508362s ub 88.526153564s int 16.000000000s))): pg present 2022-03-29T17:51:05.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519508362s ub 88.526153564s int 16.000000000s) for pg: 2.5 2022-03-29T17:51:05.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 80.519508362s ub 88.526153564s int 16.000000000s) 2022-03-29T17:51:05.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 88.440498352s 2022-03-29T17:51:05.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 88.440498352s 2022-03-29T17:51:05.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519508362s ub 88.526153564s int 16.000000000s))): submitting ctx 2022-03-29T17:51:05.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,831 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:51:05.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 80.519996643s 2022-03-29T17:51:05.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.519996643s (sent 88.528106689s) 2022-03-29T17:51:05.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519508362s ub 88.526153564s int 16.000000000s))): complete 2022-03-29T17:51:05.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:05.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:51:05.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 80.521705627s 2022-03-29T17:51:05.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.521705627s (sent 88.528640747s) 2022-03-29T17:51:05.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:05.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,832 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:51:05.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 80.519104004s 2022-03-29T17:51:05.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.519104004s (sent 88.529022217s) 2022-03-29T17:51:05.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:05.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:51:05.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.454757690s ub 88.462387085s int 16.000000000s))): start 2022-03-29T17:51:05.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:51:05.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519996643s ub 88.528106689s int 16.000000000s))): start 2022-03-29T17:51:05.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519996643s ub 88.528106689s int 16.000000000s))): got map 13 2022-03-29T17:51:05.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519996643s ub 88.528106689s int 16.000000000s))): pg present 2022-03-29T17:51:05.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519996643s ub 88.528106689s int 16.000000000s) for pg: 2.7 2022-03-29T17:51:05.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 80.519996643s ub 88.528106689s int 16.000000000s) 2022-03-29T17:51:05.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 88.443351746s 2022-03-29T17:51:05.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 88.443351746s 2022-03-29T17:51:05.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519996643s ub 88.528106689s int 16.000000000s))): submitting ctx 2022-03-29T17:51:05.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:51:05.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,834 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.521705627s ub 88.528640747s int 16.000000000s))): start 2022-03-29T17:51:05.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.521705627s ub 88.528640747s int 16.000000000s))): got map 13 2022-03-29T17:51:05.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519996643s ub 88.528106689s int 16.000000000s))): complete 2022-03-29T17:51:05.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.521705627s ub 88.528640747s int 16.000000000s))): pg present 2022-03-29T17:51:05.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:51:05.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519104004s ub 88.529022217s int 16.000000000s))): start 2022-03-29T17:51:05.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.521705627s ub 88.528640747s int 16.000000000s) for pg: 2.6 2022-03-29T17:51:05.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 80.521705627s ub 88.528640747s int 16.000000000s) 2022-03-29T17:51:05.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 88.444450378s 2022-03-29T17:51:05.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 88.444450378s 2022-03-29T17:51:05.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.521705627s ub 88.528640747s int 16.000000000s))): submitting ctx 2022-03-29T17:51:05.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519104004s ub 88.529022217s int 16.000000000s))): got map 13 2022-03-29T17:51:05.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519104004s ub 88.529022217s int 16.000000000s))): pg present 2022-03-29T17:51:05.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,835 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519104004s ub 88.529022217s int 16.000000000s) for pg: 2.4 2022-03-29T17:51:05.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 80.519104004s ub 88.529022217s int 16.000000000s) 2022-03-29T17:51:05.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 88.444885254s 2022-03-29T17:51:05.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 88.444885254s 2022-03-29T17:51:05.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519104004s ub 88.529022217s int 16.000000000s))): submitting ctx 2022-03-29T17:51:05.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.521705627s ub 88.528640747s int 16.000000000s))): complete 2022-03-29T17:51:05.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,833 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:05.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:05.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.454757690s ub 88.462387085s int 16.000000000s))): got map 13 2022-03-29T17:51:05.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:05.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,836 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.454757690s ub 88.462387085s int 16.000000000s))): pg present 2022-03-29T17:51:05.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:05.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.454757690s ub 88.462387085s int 16.000000000s) for pg: 2.0 2022-03-29T17:51:05.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 80.454757690s ub 88.462387085s int 16.000000000s) 2022-03-29T17:51:05.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 88.533340454s 2022-03-29T17:51:05.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 88.533340454s 2022-03-29T17:51:05.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.454757690s ub 88.462387085s int 16.000000000s))): submitting ctx 2022-03-29T17:51:05.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:51:05.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,837 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.526153564s))): start 2022-03-29T17:51:05.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.526153564s))): got map 13 2022-03-29T17:51:05.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.526153564s))): pg present 2022-03-29T17:51:05.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.526153564s) for pg: 2.5 2022-03-29T17:51:05.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 88.526153564s 2022-03-29T17:51:05.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.526153564s (sent 88.526153564s) 2022-03-29T17:51:05.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.526153564s))): submitting ctx 2022-03-29T17:51:05.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 80.519104004s ub 88.529022217s int 16.000000000s))): complete 2022-03-29T17:51:05.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,838 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.454757690s ub 88.462387085s int 16.000000000s))): complete 2022-03-29T17:51:05.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:51:05.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528106689s))): start 2022-03-29T17:51:05.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:05.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:05.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:51:05.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 80.464958191s 2022-03-29T17:51:05.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.464958191s (sent 88.472572327s) 2022-03-29T17:51:05.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:05.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,839 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:51:05.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.462387085s))): start 2022-03-29T17:51:05.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.462387085s))): got map 13 2022-03-29T17:51:05.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.462387085s))): pg present 2022-03-29T17:51:05.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.462387085s) for pg: 2.0 2022-03-29T17:51:05.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 88.462387085s 2022-03-29T17:51:05.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.462387085s (sent 88.462387085s) 2022-03-29T17:51:05.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.462387085s))): submitting ctx 2022-03-29T17:51:05.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,840 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,841 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528106689s))): got map 13 2022-03-29T17:51:05.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,841 [shard 0] osd - peering_event(id=59, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:05.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,841 [shard 0] osd - peering_event(id=60, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.462387085s))): complete 2022-03-29T17:51:05.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,841 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:51:05.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,841 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.464958191s ub 88.472572327s int 16.000000000s))): start 2022-03-29T17:51:05.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528106689s))): pg present 2022-03-29T17:51:05.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.526153564s))): complete 2022-03-29T17:51:05.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.464958191s ub 88.472572327s int 16.000000000s))): got map 13 2022-03-29T17:51:05.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.464958191s ub 88.472572327s int 16.000000000s))): pg present 2022-03-29T17:51:05.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528106689s) for pg: 2.7 2022-03-29T17:51:05.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 88.528106689s 2022-03-29T17:51:05.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.528106689s (sent 88.528106689s) 2022-03-29T17:51:05.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528106689s))): submitting ctx 2022-03-29T17:51:05.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,842 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.464958191s ub 88.472572327s int 16.000000000s) for pg: 2.1 2022-03-29T17:51:05.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 80.464958191s ub 88.472572327s int 16.000000000s) 2022-03-29T17:51:05.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 88.539039612s 2022-03-29T17:51:05.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 88.539039612s 2022-03-29T17:51:05.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.464958191s ub 88.472572327s int 16.000000000s))): submitting ctx 2022-03-29T17:51:05.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528106689s))): complete 2022-03-29T17:51:05.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:51:05.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528640747s))): start 2022-03-29T17:51:05.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 80.464958191s ub 88.472572327s int 16.000000000s))): complete 2022-03-29T17:51:05.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528640747s))): got map 13 2022-03-29T17:51:05.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,843 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528640747s))): pg present 2022-03-29T17:51:05.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528640747s) for pg: 2.6 2022-03-29T17:51:05.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 88.528640747s 2022-03-29T17:51:05.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.528640747s (sent 88.528640747s) 2022-03-29T17:51:05.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528640747s))): submitting ctx 2022-03-29T17:51:05.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:51:05.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.529022217s))): start 2022-03-29T17:51:05.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.529022217s))): got map 13 2022-03-29T17:51:05.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.528640747s))): complete 2022-03-29T17:51:05.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.529022217s))): pg present 2022-03-29T17:51:05.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.529022217s) for pg: 2.4 2022-03-29T17:51:05.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 88.529022217s 2022-03-29T17:51:05.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.529022217s (sent 88.529022217s) 2022-03-29T17:51:05.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.529022217s))): submitting ctx 2022-03-29T17:51:05.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,844 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,845 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 88.529022217s))): complete 2022-03-29T17:51:05.893 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:05.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:05.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:51:05.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 80.466499329s 2022-03-29T17:51:05.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 80.466499329s (sent 88.473075867s) 2022-03-29T17:51:05.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:05.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,864 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:05.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:51:05.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 80.466499329s ub 88.473075867s int 16.000000000s))): start 2022-03-29T17:51:05.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 80.466499329s ub 88.473075867s int 16.000000000s))): got map 13 2022-03-29T17:51:05.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 80.466499329s ub 88.473075867s int 16.000000000s))): pg present 2022-03-29T17:51:05.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 80.466499329s ub 88.473075867s int 16.000000000s) for pg: 2.2 2022-03-29T17:51:05.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 80.466499329s ub 88.473075867s int 16.000000000s) 2022-03-29T17:51:05.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 88.561500549s 2022-03-29T17:51:05.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 88.561500549s 2022-03-29T17:51:05.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 80.466499329s ub 88.473075867s int 16.000000000s))): submitting ctx 2022-03-29T17:51:05.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,865 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:05,866 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 80.466499329s ub 88.473075867s int 16.000000000s))): complete 2022-03-29T17:51:05.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:51:05.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.472572327s))): start 2022-03-29T17:51:05.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.472572327s))): got map 13 2022-03-29T17:51:05.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.472572327s))): pg present 2022-03-29T17:51:05.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.472572327s) for pg: 2.1 2022-03-29T17:51:05.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 88.472572327s 2022-03-29T17:51:05.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.472572327s (sent 88.472572327s) 2022-03-29T17:51:05.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.903 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.472572327s))): submitting ctx 2022-03-29T17:51:05.903 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,882 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.903 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:05,883 [shard 0] osd - peering_event(id=61, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.472572327s))): complete 2022-03-29T17:51:05.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:51:05.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.473075867s))): start 2022-03-29T17:51:05.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.473075867s))): got map 13 2022-03-29T17:51:05.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.473075867s))): pg present 2022-03-29T17:51:05.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.473075867s) for pg: 2.2 2022-03-29T17:51:05.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 88.473075867s 2022-03-29T17:51:05.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.473075867s (sent 88.473075867s) 2022-03-29T17:51:05.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - log is not dirty 2022-03-29T17:51:05.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.473075867s))): submitting ctx 2022-03-29T17:51:05.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,905 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:05.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:05,906 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 88.473075867s))): complete 2022-03-29T17:51:06.588 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:06,587 [shard 0] alienstore - stat 2022-03-29T17:51:06.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:06,603 [shard 0] alienstore - stat 2022-03-29T17:51:06.605 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:06,604 [shard 0] alienstore - stat 2022-03-29T17:51:07.593 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:07,592 [shard 0] alienstore - stat 2022-03-29T17:51:07.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:07,607 [shard 0] alienstore - stat 2022-03-29T17:51:07.610 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:07,609 [shard 0] alienstore - stat 2022-03-29T17:51:08.598 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:08,597 [shard 0] alienstore - stat 2022-03-29T17:51:08.613 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:08,612 [shard 0] alienstore - stat 2022-03-29T17:51:08.615 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:08,614 [shard 0] alienstore - stat 2022-03-29T17:51:09.603 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:09,601 [shard 0] alienstore - stat 2022-03-29T17:51:09.618 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:09,616 [shard 0] alienstore - stat 2022-03-29T17:51:09.620 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:09,619 [shard 0] alienstore - stat 2022-03-29T17:51:10.606 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:10,604 [shard 0] alienstore - stat 2022-03-29T17:51:10.621 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:10,619 [shard 0] alienstore - stat 2022-03-29T17:51:10.624 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:10,623 [shard 0] alienstore - stat 2022-03-29T17:51:10.796 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,793 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:51:10.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,793 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:51:10.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,793 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:51:10.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,793 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 85.481430054s 2022-03-29T17:51:10.797 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,793 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 85.481430054s (sent 93.489669800s) 2022-03-29T17:51:10.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,793 [shard 0] osd - log is not dirty 2022-03-29T17:51:10.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,793 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:51:10.798 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,793 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:10.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,794 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:51:10.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,794 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 85.481430054s ub 93.489669800s int 16.000000000s))): start 2022-03-29T17:51:10.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 85.481430054s ub 93.489669800s int 16.000000000s))): got map 13 2022-03-29T17:51:10.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 85.481430054s ub 93.489669800s int 16.000000000s))): pg present 2022-03-29T17:51:10.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 85.481430054s ub 93.489669800s int 16.000000000s) for pg: 1.0 2022-03-29T17:51:10.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 85.481430054s ub 93.489669800s int 16.000000000s) 2022-03-29T17:51:10.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 93.404060364s 2022-03-29T17:51:10.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 93.404060364s 2022-03-29T17:51:10.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - log is not dirty 2022-03-29T17:51:10.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 85.481430054s ub 93.489669800s int 16.000000000s))): submitting ctx 2022-03-29T17:51:10.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:10.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,795 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:51:10.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:51:10.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 93.489669800s))): start 2022-03-29T17:51:10.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 85.481430054s ub 93.489669800s int 16.000000000s))): complete 2022-03-29T17:51:10.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 93.489669800s))): got map 13 2022-03-29T17:51:10.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 93.489669800s))): pg present 2022-03-29T17:51:10.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 93.489669800s) for pg: 1.0 2022-03-29T17:51:10.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 93.489669800s 2022-03-29T17:51:10.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 93.489669800s (sent 93.489669800s) 2022-03-29T17:51:10.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - log is not dirty 2022-03-29T17:51:10.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 93.489669800s))): submitting ctx 2022-03-29T17:51:10.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,796 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:10.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:10,797 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 93.489669800s))): complete 2022-03-29T17:51:11.610 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:11,608 [shard 0] alienstore - stat 2022-03-29T17:51:11.625 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:11,623 [shard 0] alienstore - stat 2022-03-29T17:51:11.628 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:11,627 [shard 0] alienstore - stat 2022-03-29T17:51:12.614 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:12,612 [shard 0] alienstore - stat 2022-03-29T17:51:12.629 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:12,627 [shard 0] alienstore - stat 2022-03-29T17:51:12.632 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:12,631 [shard 0] alienstore - stat 2022-03-29T17:51:13.619 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:13,618 [shard 0] alienstore - stat 2022-03-29T17:51:13.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:13,633 [shard 0] alienstore - stat 2022-03-29T17:51:13.637 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:13,636 [shard 0] alienstore - stat 2022-03-29T17:51:13.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,816 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:13.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,817 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:13.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,817 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:51:13.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,817 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 88.505439758s 2022-03-29T17:51:13.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,817 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.505439758s (sent 96.513122559s) 2022-03-29T17:51:13.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,817 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,817 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:13.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,817 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.822 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,818 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:51:13.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,818 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.505439758s ub 96.513122559s int 16.000000000s))): start 2022-03-29T17:51:13.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,818 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.505439758s ub 96.513122559s int 16.000000000s))): got map 13 2022-03-29T17:51:13.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,818 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.505439758s ub 96.513122559s int 16.000000000s))): pg present 2022-03-29T17:51:13.823 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,818 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.505439758s ub 96.513122559s int 16.000000000s) for pg: 2.3 2022-03-29T17:51:13.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,818 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 88.505439758s ub 96.513122559s int 16.000000000s) 2022-03-29T17:51:13.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,819 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 96.452018738s 2022-03-29T17:51:13.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,819 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 96.452018738s 2022-03-29T17:51:13.824 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,819 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.825 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,819 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.505439758s ub 96.513122559s int 16.000000000s))): submitting ctx 2022-03-29T17:51:13.825 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,819 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,819 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:13.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,820 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:51:13.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,820 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 96.513122559s))): start 2022-03-29T17:51:13.826 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,820 [shard 0] osd - peering_event(id=62, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.505439758s ub 96.513122559s int 16.000000000s))): complete 2022-03-29T17:51:13.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,821 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 96.513122559s))): got map 13 2022-03-29T17:51:13.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,821 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 96.513122559s))): pg present 2022-03-29T17:51:13.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,821 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 96.513122559s) for pg: 2.3 2022-03-29T17:51:13.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,821 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 96.513122559s 2022-03-29T17:51:13.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,821 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.513122559s (sent 96.513122559s) 2022-03-29T17:51:13.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,821 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,821 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 96.513122559s))): submitting ctx 2022-03-29T17:51:13.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,821 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,822 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 96.513122559s))): complete 2022-03-29T17:51:13.837 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,834 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:13.837 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,834 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:13.838 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,834 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:51:13.838 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,834 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 88.462387085s 2022-03-29T17:51:13.838 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,834 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.462387085s (sent 96.467735291s) 2022-03-29T17:51:13.838 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,834 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.839 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,834 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:13.839 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,834 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,835 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:51:13.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,835 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.462387085s ub 96.467735291s int 16.000000000s))): start 2022-03-29T17:51:13.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - peering_event(id=63, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:13.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,835 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.462387085s ub 96.467735291s int 16.000000000s))): got map 13 2022-03-29T17:51:13.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.462387085s ub 96.467735291s int 16.000000000s))): pg present 2022-03-29T17:51:13.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.462387085s ub 96.467735291s int 16.000000000s) for pg: 2.0 2022-03-29T17:51:13.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 88.462387085s ub 96.467735291s int 16.000000000s) 2022-03-29T17:51:13.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 96.532592773s 2022-03-29T17:51:13.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 96.532592773s 2022-03-29T17:51:13.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.462387085s ub 96.467735291s int 16.000000000s))): submitting ctx 2022-03-29T17:51:13.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,836 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:13.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:13.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:13.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:13.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:13.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:13.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:13.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:13.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:51:13.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 88.528106689s 2022-03-29T17:51:13.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.528106689s (sent 96.533676147s) 2022-03-29T17:51:13.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:13.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,837 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.462387085s ub 96.467735291s int 16.000000000s))): complete 2022-03-29T17:51:13.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:51:13.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 88.529022217s 2022-03-29T17:51:13.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.529022217s (sent 96.534034729s) 2022-03-29T17:51:13.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:13.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:51:13.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 88.526153564s 2022-03-29T17:51:13.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.526153564s (sent 96.534439087s) 2022-03-29T17:51:13.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:13.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:51:13.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528106689s ub 96.533676147s int 16.000000000s))): start 2022-03-29T17:51:13.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528106689s ub 96.533676147s int 16.000000000s))): got map 13 2022-03-29T17:51:13.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528106689s ub 96.533676147s int 16.000000000s))): pg present 2022-03-29T17:51:13.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528106689s ub 96.533676147s int 16.000000000s) for pg: 2.7 2022-03-29T17:51:13.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 88.528106689s ub 96.533676147s int 16.000000000s) 2022-03-29T17:51:13.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 96.448699951s 2022-03-29T17:51:13.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 96.448699951s 2022-03-29T17:51:13.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,839 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,840 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528106689s ub 96.533676147s int 16.000000000s))): submitting ctx 2022-03-29T17:51:13.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,840 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,841 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528106689s ub 96.533676147s int 16.000000000s))): complete 2022-03-29T17:51:13.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,841 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:51:13.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,841 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.529022217s ub 96.534034729s int 16.000000000s))): start 2022-03-29T17:51:13.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,841 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.529022217s ub 96.534034729s int 16.000000000s))): got map 13 2022-03-29T17:51:13.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,841 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.529022217s ub 96.534034729s int 16.000000000s))): pg present 2022-03-29T17:51:13.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:51:13.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.526153564s ub 96.534439087s int 16.000000000s))): start 2022-03-29T17:51:13.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.529022217s ub 96.534034729s int 16.000000000s) for pg: 2.4 2022-03-29T17:51:13.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 88.529022217s ub 96.534034729s int 16.000000000s) 2022-03-29T17:51:13.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 96.451057434s 2022-03-29T17:51:13.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 96.451057434s 2022-03-29T17:51:13.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.529022217s ub 96.534034729s int 16.000000000s))): submitting ctx 2022-03-29T17:51:13.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.526153564s ub 96.534439087s int 16.000000000s))): got map 13 2022-03-29T17:51:13.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.526153564s ub 96.534439087s int 16.000000000s))): pg present 2022-03-29T17:51:13.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.526153564s ub 96.534439087s int 16.000000000s) for pg: 2.5 2022-03-29T17:51:13.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 88.526153564s ub 96.534439087s int 16.000000000s) 2022-03-29T17:51:13.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 96.451713562s 2022-03-29T17:51:13.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,842 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 96.451713562s 2022-03-29T17:51:13.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.526153564s ub 96.534439087s int 16.000000000s))): submitting ctx 2022-03-29T17:51:13.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.529022217s ub 96.534034729s int 16.000000000s))): complete 2022-03-29T17:51:13.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:51:13.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 88.528640747s 2022-03-29T17:51:13.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.528640747s (sent 96.539421082s) 2022-03-29T17:51:13.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:13.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.526153564s ub 96.534439087s int 16.000000000s))): complete 2022-03-29T17:51:13.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,843 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:13.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:13.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:13.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:13.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:51:13.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 88.472572327s 2022-03-29T17:51:13.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.472572327s (sent 96.477645874s) 2022-03-29T17:51:13.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:13.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,844 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:51:13.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.467735291s))): start 2022-03-29T17:51:13.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.467735291s))): got map 13 2022-03-29T17:51:13.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - peering_event(id=64, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:13.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.467735291s))): pg present 2022-03-29T17:51:13.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.467735291s) for pg: 2.0 2022-03-29T17:51:13.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 96.467735291s 2022-03-29T17:51:13.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.467735291s (sent 96.467735291s) 2022-03-29T17:51:13.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.467735291s))): submitting ctx 2022-03-29T17:51:13.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,845 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,846 [shard 0] osd - peering_event(id=65, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.467735291s))): complete 2022-03-29T17:51:13.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,846 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:51:13.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,846 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528640747s ub 96.539421082s int 16.000000000s))): start 2022-03-29T17:51:13.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528640747s ub 96.539421082s int 16.000000000s))): got map 13 2022-03-29T17:51:13.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528640747s ub 96.539421082s int 16.000000000s))): pg present 2022-03-29T17:51:13.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528640747s ub 96.539421082s int 16.000000000s) for pg: 2.6 2022-03-29T17:51:13.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 88.528640747s ub 96.539421082s int 16.000000000s) 2022-03-29T17:51:13.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 96.456298828s 2022-03-29T17:51:13.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 96.456298828s 2022-03-29T17:51:13.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528640747s ub 96.539421082s int 16.000000000s))): submitting ctx 2022-03-29T17:51:13.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,847 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,848 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:13.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,848 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:13.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,848 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 88.528640747s ub 96.539421082s int 16.000000000s))): complete 2022-03-29T17:51:13.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,848 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:51:13.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,848 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.472572327s ub 96.477645874s int 16.000000000s))): start 2022-03-29T17:51:13.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,849 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.472572327s ub 96.477645874s int 16.000000000s))): got map 13 2022-03-29T17:51:13.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,849 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.472572327s ub 96.477645874s int 16.000000000s))): pg present 2022-03-29T17:51:13.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,849 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:51:13.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,849 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.533676147s))): start 2022-03-29T17:51:13.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,849 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.472572327s ub 96.477645874s int 16.000000000s) for pg: 2.1 2022-03-29T17:51:13.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,850 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 88.472572327s ub 96.477645874s int 16.000000000s) 2022-03-29T17:51:13.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,850 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 96.545959473s 2022-03-29T17:51:13.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,850 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 96.545959473s 2022-03-29T17:51:13.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,850 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,850 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.472572327s ub 96.477645874s int 16.000000000s))): submitting ctx 2022-03-29T17:51:13.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,850 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.533676147s))): got map 13 2022-03-29T17:51:13.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 88.472572327s ub 96.477645874s int 16.000000000s))): complete 2022-03-29T17:51:13.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.533676147s))): pg present 2022-03-29T17:51:13.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.533676147s) for pg: 2.7 2022-03-29T17:51:13.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 96.533676147s 2022-03-29T17:51:13.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.533676147s (sent 96.533676147s) 2022-03-29T17:51:13.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.533676147s))): submitting ctx 2022-03-29T17:51:13.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,852 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:51:13.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,852 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534034729s))): start 2022-03-29T17:51:13.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,852 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.533676147s))): complete 2022-03-29T17:51:13.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,852 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534034729s))): got map 13 2022-03-29T17:51:13.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,852 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:51:13.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534439087s))): start 2022-03-29T17:51:13.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534034729s))): pg present 2022-03-29T17:51:13.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534439087s))): got map 13 2022-03-29T17:51:13.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534439087s))): pg present 2022-03-29T17:51:13.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534034729s) for pg: 2.4 2022-03-29T17:51:13.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 96.534034729s 2022-03-29T17:51:13.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.534034729s (sent 96.534034729s) 2022-03-29T17:51:13.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534034729s))): submitting ctx 2022-03-29T17:51:13.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534439087s) for pg: 2.5 2022-03-29T17:51:13.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,853 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 96.534439087s 2022-03-29T17:51:13.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,854 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.534439087s (sent 96.534439087s) 2022-03-29T17:51:13.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,854 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,854 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534439087s))): submitting ctx 2022-03-29T17:51:13.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,854 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,854 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:51:13.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,854 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.539421082s))): start 2022-03-29T17:51:13.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,854 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534439087s))): complete 2022-03-29T17:51:13.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,854 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.539421082s))): got map 13 2022-03-29T17:51:13.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.534034729s))): complete 2022-03-29T17:51:13.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.539421082s))): pg present 2022-03-29T17:51:13.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.539421082s) for pg: 2.6 2022-03-29T17:51:13.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 96.539421082s 2022-03-29T17:51:13.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.539421082s (sent 96.539421082s) 2022-03-29T17:51:13.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.539421082s))): submitting ctx 2022-03-29T17:51:13.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,855 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 96.539421082s))): complete 2022-03-29T17:51:13.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,869 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:13.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,869 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:13.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,869 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:51:13.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,870 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 88.473075867s 2022-03-29T17:51:13.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,870 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 88.473075867s (sent 96.478881836s) 2022-03-29T17:51:13.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,870 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,870 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:13.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,870 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,870 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:13.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:51:13.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 88.473075867s ub 96.478881836s int 16.000000000s))): start 2022-03-29T17:51:13.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 88.473075867s ub 96.478881836s int 16.000000000s))): got map 13 2022-03-29T17:51:13.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 88.473075867s ub 96.478881836s int 16.000000000s))): pg present 2022-03-29T17:51:13.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 88.473075867s ub 96.478881836s int 16.000000000s) for pg: 2.2 2022-03-29T17:51:13.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 88.473075867s ub 96.478881836s int 16.000000000s) 2022-03-29T17:51:13.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 96.567573547s 2022-03-29T17:51:13.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 96.567573547s 2022-03-29T17:51:13.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 88.473075867s ub 96.478881836s int 16.000000000s))): submitting ctx 2022-03-29T17:51:13.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,871 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:13,872 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 88.473075867s ub 96.478881836s int 16.000000000s))): complete 2022-03-29T17:51:13.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,889 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:51:13.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,889 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.477645874s))): start 2022-03-29T17:51:13.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,889 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.477645874s))): got map 13 2022-03-29T17:51:13.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,889 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.477645874s))): pg present 2022-03-29T17:51:13.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,890 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.477645874s) for pg: 2.1 2022-03-29T17:51:13.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,890 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 96.477645874s 2022-03-29T17:51:13.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,890 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.477645874s (sent 96.477645874s) 2022-03-29T17:51:13.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,890 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,890 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.477645874s))): submitting ctx 2022-03-29T17:51:13.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,890 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:13,891 [shard 0] osd - peering_event(id=66, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.477645874s))): complete 2022-03-29T17:51:13.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,911 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:51:13.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,911 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.478881836s))): start 2022-03-29T17:51:13.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,911 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.478881836s))): got map 13 2022-03-29T17:51:13.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,911 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.478881836s))): pg present 2022-03-29T17:51:13.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,911 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.478881836s) for pg: 2.2 2022-03-29T17:51:13.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,912 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 96.478881836s 2022-03-29T17:51:13.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,912 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.478881836s (sent 96.478881836s) 2022-03-29T17:51:13.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,912 [shard 0] osd - log is not dirty 2022-03-29T17:51:13.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,912 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.478881836s))): submitting ctx 2022-03-29T17:51:13.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,912 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:13.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:13,912 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 96.478881836s))): complete 2022-03-29T17:51:14.628 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:14,624 [shard 0] alienstore - stat 2022-03-29T17:51:14.640 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:14,639 [shard 0] alienstore - stat 2022-03-29T17:51:14.643 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:14,642 [shard 0] alienstore - stat 2022-03-29T17:51:15.631 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:15,629 [shard 0] alienstore - stat 2022-03-29T17:51:15.645 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:15,643 [shard 0] alienstore - stat 2022-03-29T17:51:15.648 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:15,647 [shard 0] alienstore - stat 2022-03-29T17:51:16.635 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:16,634 [shard 0] alienstore - stat 2022-03-29T17:51:16.649 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:16,648 [shard 0] alienstore - stat 2022-03-29T17:51:16.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:16,651 [shard 0] alienstore - stat 2022-03-29T17:51:17.640 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:17,638 [shard 0] alienstore - stat 2022-03-29T17:51:17.653 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:17,652 [shard 0] alienstore - stat 2022-03-29T17:51:17.656 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:17,655 [shard 0] alienstore - stat 2022-03-29T17:51:18.645 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:18,643 [shard 0] alienstore - stat 2022-03-29T17:51:18.658 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:18,657 [shard 0] alienstore - stat 2022-03-29T17:51:18.661 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:18,659 [shard 0] alienstore - stat 2022-03-29T17:51:18.802 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,800 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:51:18.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,800 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:51:18.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,800 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:51:18.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,800 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 93.489669800s 2022-03-29T17:51:18.803 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,800 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 93.489669800s (sent 101.496574402s) 2022-03-29T17:51:18.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,800 [shard 0] osd - log is not dirty 2022-03-29T17:51:18.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,800 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:51:18.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,800 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:18.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,801 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:51:18.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,801 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 93.489669800s ub 101.496574402s int 16.000000000s))): start 2022-03-29T17:51:18.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,801 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 93.489669800s ub 101.496574402s int 16.000000000s))): got map 13 2022-03-29T17:51:18.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,801 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 93.489669800s ub 101.496574402s int 16.000000000s))): pg present 2022-03-29T17:51:18.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,801 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 93.489669800s ub 101.496574402s int 16.000000000s) for pg: 1.0 2022-03-29T17:51:18.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,801 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 93.489669800s ub 101.496574402s int 16.000000000s) 2022-03-29T17:51:18.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,802 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 101.410873413s 2022-03-29T17:51:18.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,802 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 101.410873413s 2022-03-29T17:51:18.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,802 [shard 0] osd - log is not dirty 2022-03-29T17:51:18.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,802 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 93.489669800s ub 101.496574402s int 16.000000000s))): submitting ctx 2022-03-29T17:51:18.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,802 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:18.809 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,802 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:51:18.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:18,802 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 93.489669800s ub 101.496574402s int 16.000000000s))): complete 2022-03-29T17:51:18.811 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:51:18.811 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 101.496574402s))): start 2022-03-29T17:51:18.811 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 101.496574402s))): got map 13 2022-03-29T17:51:18.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 101.496574402s))): pg present 2022-03-29T17:51:18.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 101.496574402s) for pg: 1.0 2022-03-29T17:51:18.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 101.496574402s 2022-03-29T17:51:18.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 101.496574402s (sent 101.496574402s) 2022-03-29T17:51:18.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - log is not dirty 2022-03-29T17:51:18.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 101.496574402s))): submitting ctx 2022-03-29T17:51:18.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,803 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:18.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:18,804 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 101.496574402s))): complete 2022-03-29T17:51:19.650 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:19,648 [shard 0] alienstore - stat 2022-03-29T17:51:19.662 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:19,661 [shard 0] alienstore - stat 2022-03-29T17:51:19.665 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:19,664 [shard 0] alienstore - stat 2022-03-29T17:51:20.654 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:20,652 [shard 0] alienstore - stat 2022-03-29T17:51:20.667 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:20,666 [shard 0] alienstore - stat 2022-03-29T17:51:20.670 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:20,669 [shard 0] alienstore - stat 2022-03-29T17:51:21.659 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:21,657 [shard 0] alienstore - stat 2022-03-29T17:51:21.672 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:21,671 [shard 0] alienstore - stat 2022-03-29T17:51:21.675 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:21,673 [shard 0] alienstore - stat 2022-03-29T17:51:21.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,824 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:21.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,824 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:21.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,824 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:51:21.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,824 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 96.513122559s 2022-03-29T17:51:21.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,824 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.513122559s (sent 104.520591736s) 2022-03-29T17:51:21.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,824 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,824 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:21.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,824 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.829 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,825 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:51:21.830 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,825 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.513122559s ub 104.520591736s int 16.000000000s))): start 2022-03-29T17:51:21.830 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,825 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.513122559s ub 104.520591736s int 16.000000000s))): got map 13 2022-03-29T17:51:21.830 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,825 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.513122559s ub 104.520591736s int 16.000000000s))): pg present 2022-03-29T17:51:21.830 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,825 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.513122559s ub 104.520591736s int 16.000000000s) for pg: 2.3 2022-03-29T17:51:21.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,825 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 96.513122559s ub 104.520591736s int 16.000000000s) 2022-03-29T17:51:21.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,826 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 104.459045410s 2022-03-29T17:51:21.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,826 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 104.459045410s 2022-03-29T17:51:21.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,826 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,826 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.513122559s ub 104.520591736s int 16.000000000s))): submitting ctx 2022-03-29T17:51:21.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,826 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,826 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:21.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,826 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:51:21.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,826 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 104.520591736s))): start 2022-03-29T17:51:21.834 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - peering_event(id=67, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.513122559s ub 104.520591736s int 16.000000000s))): complete 2022-03-29T17:51:21.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 104.520591736s))): got map 13 2022-03-29T17:51:21.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 104.520591736s))): pg present 2022-03-29T17:51:21.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 104.520591736s) for pg: 2.3 2022-03-29T17:51:21.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 104.520591736s 2022-03-29T17:51:21.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.520591736s (sent 104.520591736s) 2022-03-29T17:51:21.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 104.520591736s))): submitting ctx 2022-03-29T17:51:21.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,827 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,828 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 104.520591736s))): complete 2022-03-29T17:51:21.844 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:21.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:21.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:51:21.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 96.467735291s 2022-03-29T17:51:21.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.467735291s (sent 104.475372314s) 2022-03-29T17:51:21.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:21.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,842 [shard 0] osd - peering_event(id=68, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:21.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,843 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:51:21.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,843 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.467735291s ub 104.475372314s int 16.000000000s))): start 2022-03-29T17:51:21.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,843 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.467735291s ub 104.475372314s int 16.000000000s))): got map 13 2022-03-29T17:51:21.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,843 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.467735291s ub 104.475372314s int 16.000000000s))): pg present 2022-03-29T17:51:21.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,843 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.467735291s ub 104.475372314s int 16.000000000s) for pg: 2.0 2022-03-29T17:51:21.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,843 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 96.467735291s ub 104.475372314s int 16.000000000s) 2022-03-29T17:51:21.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,843 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 104.539535522s 2022-03-29T17:51:21.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,843 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 104.539535522s 2022-03-29T17:51:21.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.467735291s ub 104.475372314s int 16.000000000s))): submitting ctx 2022-03-29T17:51:21.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:21.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:21.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:21.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:21.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:21.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:21.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:21.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:21.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.467735291s ub 104.475372314s int 16.000000000s))): complete 2022-03-29T17:51:21.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:51:21.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 96.539421082s 2022-03-29T17:51:21.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,844 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.539421082s (sent 104.540847778s) 2022-03-29T17:51:21.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:21.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:51:21.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 96.533676147s 2022-03-29T17:51:21.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.533676147s (sent 104.541175842s) 2022-03-29T17:51:21.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:21.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:51:21.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.539421082s ub 104.540847778s int 16.000000000s))): start 2022-03-29T17:51:21.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,845 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.539421082s ub 104.540847778s int 16.000000000s))): got map 13 2022-03-29T17:51:21.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.539421082s ub 104.540847778s int 16.000000000s))): pg present 2022-03-29T17:51:21.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:51:21.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.533676147s ub 104.541175842s int 16.000000000s))): start 2022-03-29T17:51:21.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.533676147s ub 104.541175842s int 16.000000000s))): got map 13 2022-03-29T17:51:21.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.539421082s ub 104.540847778s int 16.000000000s) for pg: 2.6 2022-03-29T17:51:21.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 96.539421082s ub 104.540847778s int 16.000000000s) 2022-03-29T17:51:21.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 104.455322266s 2022-03-29T17:51:21.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 104.455322266s 2022-03-29T17:51:21.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.539421082s ub 104.540847778s int 16.000000000s))): submitting ctx 2022-03-29T17:51:21.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.533676147s ub 104.541175842s int 16.000000000s))): pg present 2022-03-29T17:51:21.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.533676147s ub 104.541175842s int 16.000000000s) for pg: 2.7 2022-03-29T17:51:21.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 96.533676147s ub 104.541175842s int 16.000000000s) 2022-03-29T17:51:21.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 104.455703735s 2022-03-29T17:51:21.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 104.455703735s 2022-03-29T17:51:21.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.533676147s ub 104.541175842s int 16.000000000s))): submitting ctx 2022-03-29T17:51:21.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,846 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,847 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:51:21.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,847 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 96.534439087s 2022-03-29T17:51:21.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,847 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.534439087s (sent 104.543304443s) 2022-03-29T17:51:21.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,847 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.539421082s ub 104.540847778s int 16.000000000s))): complete 2022-03-29T17:51:21.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,847 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.533676147s ub 104.541175842s int 16.000000000s))): complete 2022-03-29T17:51:21.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,847 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,847 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:21.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,847 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:51:21.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 96.534034729s 2022-03-29T17:51:21.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.534034729s (sent 104.544105530s) 2022-03-29T17:51:21.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:21.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:21.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:51:21.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534439087s ub 104.543304443s int 16.000000000s))): start 2022-03-29T17:51:21.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,848 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534439087s ub 104.543304443s int 16.000000000s))): got map 13 2022-03-29T17:51:21.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:51:21.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534034729s ub 104.544105530s int 16.000000000s))): start 2022-03-29T17:51:21.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534439087s ub 104.543304443s int 16.000000000s))): pg present 2022-03-29T17:51:21.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534034729s ub 104.544105530s int 16.000000000s))): got map 13 2022-03-29T17:51:21.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534034729s ub 104.544105530s int 16.000000000s))): pg present 2022-03-29T17:51:21.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534439087s ub 104.543304443s int 16.000000000s) for pg: 2.5 2022-03-29T17:51:21.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 96.534439087s ub 104.543304443s int 16.000000000s) 2022-03-29T17:51:21.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 104.458312988s 2022-03-29T17:51:21.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 104.458312988s 2022-03-29T17:51:21.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534439087s ub 104.543304443s int 16.000000000s))): submitting ctx 2022-03-29T17:51:21.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534034729s ub 104.544105530s int 16.000000000s) for pg: 2.4 2022-03-29T17:51:21.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 96.534034729s ub 104.544105530s int 16.000000000s) 2022-03-29T17:51:21.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 104.458648682s 2022-03-29T17:51:21.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 104.458648682s 2022-03-29T17:51:21.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534034729s ub 104.544105530s int 16.000000000s))): submitting ctx 2022-03-29T17:51:21.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,849 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,850 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534439087s ub 104.543304443s int 16.000000000s))): complete 2022-03-29T17:51:21.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,850 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:21.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,850 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:21.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,850 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:51:21.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,850 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.540847778s))): start 2022-03-29T17:51:21.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.540847778s))): got map 13 2022-03-29T17:51:21.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.540847778s))): pg present 2022-03-29T17:51:21.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.540847778s) for pg: 2.6 2022-03-29T17:51:21.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 104.540847778s 2022-03-29T17:51:21.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.540847778s (sent 104.540847778s) 2022-03-29T17:51:21.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.540847778s))): submitting ctx 2022-03-29T17:51:21.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:51:21.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,851 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.541175842s))): start 2022-03-29T17:51:21.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:21.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:21.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:51:21.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 96.477645874s 2022-03-29T17:51:21.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.477645874s (sent 104.485252380s) 2022-03-29T17:51:21.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:21.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:51:21.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.475372314s))): start 2022-03-29T17:51:21.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.475372314s))): got map 13 2022-03-29T17:51:21.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,852 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.475372314s))): pg present 2022-03-29T17:51:21.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,853 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.475372314s) for pg: 2.0 2022-03-29T17:51:21.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,853 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 104.475372314s 2022-03-29T17:51:21.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,853 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.475372314s (sent 104.475372314s) 2022-03-29T17:51:21.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,853 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,853 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.475372314s))): submitting ctx 2022-03-29T17:51:21.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,853 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.889 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,853 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 96.534034729s ub 104.544105530s int 16.000000000s))): complete 2022-03-29T17:51:21.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:21.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.541175842s))): got map 13 2022-03-29T17:51:21.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.541175842s))): pg present 2022-03-29T17:51:21.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:51:21.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.477645874s ub 104.485252380s int 16.000000000s))): start 2022-03-29T17:51:21.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.541175842s) for pg: 2.7 2022-03-29T17:51:21.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 104.541175842s 2022-03-29T17:51:21.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.541175842s (sent 104.541175842s) 2022-03-29T17:51:21.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.541175842s))): submitting ctx 2022-03-29T17:51:21.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.477645874s ub 104.485252380s int 16.000000000s))): got map 13 2022-03-29T17:51:21.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,854 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.477645874s ub 104.485252380s int 16.000000000s))): pg present 2022-03-29T17:51:21.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,855 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.477645874s ub 104.485252380s int 16.000000000s) for pg: 2.1 2022-03-29T17:51:21.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,855 [shard 0] osd - peering_event(id=69, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:21.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,855 [shard 0] osd - peering_event(id=70, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.475372314s))): complete 2022-03-29T17:51:21.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,855 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 96.477645874s ub 104.485252380s int 16.000000000s) 2022-03-29T17:51:21.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,855 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 104.551856995s 2022-03-29T17:51:21.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,855 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 104.551856995s 2022-03-29T17:51:21.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.477645874s ub 104.485252380s int 16.000000000s))): submitting ctx 2022-03-29T17:51:21.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.540847778s))): complete 2022-03-29T17:51:21.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.541175842s))): complete 2022-03-29T17:51:21.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:51:21.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.543304443s))): start 2022-03-29T17:51:21.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.543304443s))): got map 13 2022-03-29T17:51:21.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.543304443s))): pg present 2022-03-29T17:51:21.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.543304443s) for pg: 2.5 2022-03-29T17:51:21.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 104.543304443s 2022-03-29T17:51:21.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.543304443s (sent 104.543304443s) 2022-03-29T17:51:21.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.543304443s))): submitting ctx 2022-03-29T17:51:21.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:51:21.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,856 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.544105530s))): start 2022-03-29T17:51:21.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,857 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.544105530s))): got map 13 2022-03-29T17:51:21.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,857 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.544105530s))): pg present 2022-03-29T17:51:21.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,857 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.544105530s) for pg: 2.4 2022-03-29T17:51:21.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,857 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 104.544105530s 2022-03-29T17:51:21.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,857 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.544105530s (sent 104.544105530s) 2022-03-29T17:51:21.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,857 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,857 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.544105530s))): submitting ctx 2022-03-29T17:51:21.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,857 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,858 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 96.477645874s ub 104.485252380s int 16.000000000s))): complete 2022-03-29T17:51:21.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,858 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.543304443s))): complete 2022-03-29T17:51:21.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,858 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 104.544105530s))): complete 2022-03-29T17:51:21.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,879 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:21.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,879 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:21.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,879 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:51:21.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,879 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 96.478881836s 2022-03-29T17:51:21.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,879 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 96.478881836s (sent 104.488250732s) 2022-03-29T17:51:21.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,879 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,879 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:21.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,879 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:21.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:51:21.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 96.478881836s ub 104.488250732s int 16.000000000s))): start 2022-03-29T17:51:21.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 96.478881836s ub 104.488250732s int 16.000000000s))): got map 13 2022-03-29T17:51:21.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 96.478881836s ub 104.488250732s int 16.000000000s))): pg present 2022-03-29T17:51:21.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 96.478881836s ub 104.488250732s int 16.000000000s) for pg: 2.2 2022-03-29T17:51:21.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 96.478881836s ub 104.488250732s int 16.000000000s) 2022-03-29T17:51:21.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 104.576713562s 2022-03-29T17:51:21.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 104.576713562s 2022-03-29T17:51:21.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 96.478881836s ub 104.488250732s int 16.000000000s))): submitting ctx 2022-03-29T17:51:21.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,880 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:21,881 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 96.478881836s ub 104.488250732s int 16.000000000s))): complete 2022-03-29T17:51:21.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,895 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:51:21.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,895 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.485252380s))): start 2022-03-29T17:51:21.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,895 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.485252380s))): got map 13 2022-03-29T17:51:21.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,896 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.485252380s))): pg present 2022-03-29T17:51:21.915 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,896 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.485252380s) for pg: 2.1 2022-03-29T17:51:21.915 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,896 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 104.485252380s 2022-03-29T17:51:21.915 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,896 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.485252380s (sent 104.485252380s) 2022-03-29T17:51:21.915 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,896 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.916 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,896 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.485252380s))): submitting ctx 2022-03-29T17:51:21.916 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,896 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.916 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:21,896 [shard 0] osd - peering_event(id=71, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.485252380s))): complete 2022-03-29T17:51:21.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:51:21.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.488250732s))): start 2022-03-29T17:51:21.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.488250732s))): got map 13 2022-03-29T17:51:21.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.488250732s))): pg present 2022-03-29T17:51:21.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.488250732s) for pg: 2.2 2022-03-29T17:51:21.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 104.488250732s 2022-03-29T17:51:21.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.488250732s (sent 104.488250732s) 2022-03-29T17:51:21.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - log is not dirty 2022-03-29T17:51:21.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.488250732s))): submitting ctx 2022-03-29T17:51:21.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,920 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:21.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:21,921 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 104.488250732s))): complete 2022-03-29T17:51:22.664 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:22,663 [shard 0] alienstore - stat 2022-03-29T17:51:22.677 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:22,676 [shard 0] alienstore - stat 2022-03-29T17:51:22.680 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:22,678 [shard 0] alienstore - stat 2022-03-29T17:51:23.669 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:23,667 [shard 0] alienstore - stat 2022-03-29T17:51:23.686 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:23,684 [shard 0] alienstore - stat 2022-03-29T17:51:23.686 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:23,685 [shard 0] alienstore - stat 2022-03-29T17:51:24.674 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:24,672 [shard 0] alienstore - stat 2022-03-29T17:51:24.691 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:24,689 [shard 0] alienstore - stat 2022-03-29T17:51:24.691 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:24,689 [shard 0] alienstore - stat 2022-03-29T17:51:25.684 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:25,676 [shard 0] alienstore - stat 2022-03-29T17:51:25.695 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:25,693 [shard 0] alienstore - stat 2022-03-29T17:51:25.695 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:25,694 [shard 0] alienstore - stat 2022-03-29T17:51:26.683 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:26,681 [shard 0] alienstore - stat 2022-03-29T17:51:26.700 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:26,698 [shard 0] alienstore - stat 2022-03-29T17:51:26.700 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:26,698 [shard 0] alienstore - stat 2022-03-29T17:51:26.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,809 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:51:26.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,809 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:51:26.812 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,809 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:51:26.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,809 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 101.496574402s 2022-03-29T17:51:26.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,809 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 101.496574402s (sent 109.505271912s) 2022-03-29T17:51:26.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,809 [shard 0] osd - log is not dirty 2022-03-29T17:51:26.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,809 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:51:26.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,809 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:26.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,810 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:51:26.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,810 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 101.496574402s ub 109.505271912s int 16.000000000s))): start 2022-03-29T17:51:26.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,810 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:51:26.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 101.496574402s ub 109.505271912s int 16.000000000s))): got map 13 2022-03-29T17:51:26.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 101.496574402s ub 109.505271912s int 16.000000000s))): pg present 2022-03-29T17:51:26.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 101.496574402s ub 109.505271912s int 16.000000000s) for pg: 1.0 2022-03-29T17:51:26.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 101.496574402s ub 109.505271912s int 16.000000000s) 2022-03-29T17:51:26.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 109.420082092s 2022-03-29T17:51:26.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 109.420082092s 2022-03-29T17:51:26.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - log is not dirty 2022-03-29T17:51:26.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 101.496574402s ub 109.505271912s int 16.000000000s))): submitting ctx 2022-03-29T17:51:26.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:26.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:51:26.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 109.505271912s))): start 2022-03-29T17:51:26.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 109.505271912s))): got map 13 2022-03-29T17:51:26.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,811 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 109.505271912s))): pg present 2022-03-29T17:51:26.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,812 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 109.505271912s) for pg: 1.0 2022-03-29T17:51:26.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,812 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 109.505271912s 2022-03-29T17:51:26.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,812 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 109.505271912s (sent 109.505271912s) 2022-03-29T17:51:26.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,812 [shard 0] osd - log is not dirty 2022-03-29T17:51:26.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,812 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 109.505271912s))): submitting ctx 2022-03-29T17:51:26.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,812 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:26.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:26,812 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 109.505271912s))): complete 2022-03-29T17:51:26.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:26,812 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 101.496574402s ub 109.505271912s int 16.000000000s))): complete 2022-03-29T17:51:27.688 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:27,686 [shard 0] alienstore - stat 2022-03-29T17:51:27.704 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:27,703 [shard 0] alienstore - stat 2022-03-29T17:51:27.705 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:27,703 [shard 0] alienstore - stat 2022-03-29T17:51:28.693 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:28,691 [shard 0] alienstore - stat 2022-03-29T17:51:28.709 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:28,708 [shard 0] alienstore - stat 2022-03-29T17:51:28.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:28,708 [shard 0] alienstore - stat 2022-03-29T17:51:29.698 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:29,697 [shard 0] alienstore - stat 2022-03-29T17:51:29.715 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:29,713 [shard 0] alienstore - stat 2022-03-29T17:51:29.715 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:29,714 [shard 0] alienstore - stat 2022-03-29T17:51:29.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,824 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:29.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,824 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:29.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,824 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:51:29.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,824 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 104.520591736s 2022-03-29T17:51:29.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,824 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.520591736s (sent 112.520591736s) 2022-03-29T17:51:29.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,824 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,824 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:29.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,824 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,825 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:51:29.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,825 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.520591736s ub 112.520591736s int 16.000000000s))): start 2022-03-29T17:51:29.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,825 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.520591736s ub 112.520591736s int 16.000000000s))): got map 13 2022-03-29T17:51:29.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,825 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.520591736s ub 112.520591736s int 16.000000000s))): pg present 2022-03-29T17:51:29.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.520591736s ub 112.520591736s int 16.000000000s) for pg: 2.3 2022-03-29T17:51:29.832 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 104.520591736s ub 112.520591736s int 16.000000000s) 2022-03-29T17:51:29.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 112.459114075s 2022-03-29T17:51:29.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 112.459114075s 2022-03-29T17:51:29.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.833 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.520591736s ub 112.520591736s int 16.000000000s))): submitting ctx 2022-03-29T17:51:29.834 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:29.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:51:29.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,826 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 112.520591736s))): start 2022-03-29T17:51:29.836 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - peering_event(id=72, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.520591736s ub 112.520591736s int 16.000000000s))): complete 2022-03-29T17:51:29.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 112.520591736s))): got map 13 2022-03-29T17:51:29.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 112.520591736s))): pg present 2022-03-29T17:51:29.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 112.520591736s) for pg: 2.3 2022-03-29T17:51:29.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 112.520591736s 2022-03-29T17:51:29.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.520591736s (sent 112.520591736s) 2022-03-29T17:51:29.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 112.520591736s))): submitting ctx 2022-03-29T17:51:29.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,827 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,828 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 112.520591736s))): complete 2022-03-29T17:51:29.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,844 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:29.847 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,844 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:29.847 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,844 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:51:29.847 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,844 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 104.475372314s 2022-03-29T17:51:29.847 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,844 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.475372314s (sent 112.477554321s) 2022-03-29T17:51:29.848 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,844 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.848 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,844 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:29.848 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,844 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:29.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:29.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:29.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:29.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:29.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:29.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:29.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:29.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:51:29.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 104.543304443s 2022-03-29T17:51:29.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=73, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:29.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.543304443s (sent 112.541427612s) 2022-03-29T17:51:29.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:29.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,845 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:51:29.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 104.540847778s 2022-03-29T17:51:29.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.540847778s (sent 112.541984558s) 2022-03-29T17:51:29.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:29.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:51:29.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 104.541175842s 2022-03-29T17:51:29.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.541175842s (sent 112.542152405s) 2022-03-29T17:51:29.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:29.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:51:29.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 104.544105530s 2022-03-29T17:51:29.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.544105530s (sent 112.542312622s) 2022-03-29T17:51:29.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:29.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:29.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,846 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:51:29.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,847 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.475372314s ub 112.477554321s int 16.000000000s))): start 2022-03-29T17:51:29.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,847 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.475372314s ub 112.477554321s int 16.000000000s))): got map 13 2022-03-29T17:51:29.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,847 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.475372314s ub 112.477554321s int 16.000000000s))): pg present 2022-03-29T17:51:29.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,847 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:29.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,847 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.475372314s ub 112.477554321s int 16.000000000s) for pg: 2.0 2022-03-29T17:51:29.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,847 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 104.475372314s ub 112.477554321s int 16.000000000s) 2022-03-29T17:51:29.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,847 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 112.543373108s 2022-03-29T17:51:29.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,847 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 112.543373108s 2022-03-29T17:51:29.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:51:29.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.543304443s ub 112.541427612s int 16.000000000s))): start 2022-03-29T17:51:29.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.543304443s ub 112.541427612s int 16.000000000s))): got map 13 2022-03-29T17:51:29.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.475372314s ub 112.477554321s int 16.000000000s))): submitting ctx 2022-03-29T17:51:29.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.543304443s ub 112.541427612s int 16.000000000s))): pg present 2022-03-29T17:51:29.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.543304443s ub 112.541427612s int 16.000000000s) for pg: 2.5 2022-03-29T17:51:29.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 104.543304443s ub 112.541427612s int 16.000000000s) 2022-03-29T17:51:29.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 112.457664490s 2022-03-29T17:51:29.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 112.457664490s 2022-03-29T17:51:29.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:29.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,849 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:29.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,849 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.475372314s ub 112.477554321s int 16.000000000s))): complete 2022-03-29T17:51:29.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,848 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,849 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.543304443s ub 112.541427612s int 16.000000000s))): submitting ctx 2022-03-29T17:51:29.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,849 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,849 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:51:29.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,849 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.540847778s ub 112.541984558s int 16.000000000s))): start 2022-03-29T17:51:29.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,849 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.543304443s ub 112.541427612s int 16.000000000s))): complete 2022-03-29T17:51:29.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.540847778s ub 112.541984558s int 16.000000000s))): got map 13 2022-03-29T17:51:29.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.540847778s ub 112.541984558s int 16.000000000s))): pg present 2022-03-29T17:51:29.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:51:29.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.541175842s ub 112.542152405s int 16.000000000s))): start 2022-03-29T17:51:29.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.540847778s ub 112.541984558s int 16.000000000s) for pg: 2.6 2022-03-29T17:51:29.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 104.540847778s ub 112.541984558s int 16.000000000s) 2022-03-29T17:51:29.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 112.459442139s 2022-03-29T17:51:29.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 112.459442139s 2022-03-29T17:51:29.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.540847778s ub 112.541984558s int 16.000000000s))): submitting ctx 2022-03-29T17:51:29.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.541175842s ub 112.542152405s int 16.000000000s))): got map 13 2022-03-29T17:51:29.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,850 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.541175842s ub 112.542152405s int 16.000000000s))): pg present 2022-03-29T17:51:29.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:51:29.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.544105530s ub 112.542312622s int 16.000000000s))): start 2022-03-29T17:51:29.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.540847778s ub 112.541984558s int 16.000000000s))): complete 2022-03-29T17:51:29.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.541175842s ub 112.542152405s int 16.000000000s) for pg: 2.7 2022-03-29T17:51:29.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 104.541175842s ub 112.542152405s int 16.000000000s) 2022-03-29T17:51:29.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 112.460342407s 2022-03-29T17:51:29.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 112.460342407s 2022-03-29T17:51:29.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.541175842s ub 112.542152405s int 16.000000000s))): submitting ctx 2022-03-29T17:51:29.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.544105530s ub 112.542312622s int 16.000000000s))): got map 13 2022-03-29T17:51:29.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.544105530s ub 112.542312622s int 16.000000000s))): pg present 2022-03-29T17:51:29.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,851 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.544105530s ub 112.542312622s int 16.000000000s) for pg: 2.4 2022-03-29T17:51:29.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,852 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 104.544105530s ub 112.542312622s int 16.000000000s) 2022-03-29T17:51:29.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,852 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 112.460922241s 2022-03-29T17:51:29.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,852 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 112.460922241s 2022-03-29T17:51:29.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,852 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,852 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.544105530s ub 112.542312622s int 16.000000000s))): submitting ctx 2022-03-29T17:51:29.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,852 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,852 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.541175842s ub 112.542152405s int 16.000000000s))): complete 2022-03-29T17:51:29.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,852 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 104.544105530s ub 112.542312622s int 16.000000000s))): complete 2022-03-29T17:51:29.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,854 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:29.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,854 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:29.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,854 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:51:29.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,854 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 104.485252380s 2022-03-29T17:51:29.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,854 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.485252380s (sent 112.487594604s) 2022-03-29T17:51:29.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,854 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,854 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:29.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,854 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:51:29.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.477554321s))): start 2022-03-29T17:51:29.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - peering_event(id=74, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:29.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.477554321s))): got map 13 2022-03-29T17:51:29.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.477554321s))): pg present 2022-03-29T17:51:29.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.477554321s) for pg: 2.0 2022-03-29T17:51:29.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 112.477554321s 2022-03-29T17:51:29.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.477554321s (sent 112.477554321s) 2022-03-29T17:51:29.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.477554321s))): submitting ctx 2022-03-29T17:51:29.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,855 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,856 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:51:29.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,856 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.485252380s ub 112.487594604s int 16.000000000s))): start 2022-03-29T17:51:29.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,856 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.485252380s ub 112.487594604s int 16.000000000s))): got map 13 2022-03-29T17:51:29.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,856 [shard 0] osd - peering_event(id=75, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.477554321s))): complete 2022-03-29T17:51:29.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,856 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.485252380s ub 112.487594604s int 16.000000000s))): pg present 2022-03-29T17:51:29.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,857 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.485252380s ub 112.487594604s int 16.000000000s) for pg: 2.1 2022-03-29T17:51:29.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,857 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 104.485252380s ub 112.487594604s int 16.000000000s) 2022-03-29T17:51:29.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,857 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 112.553115845s 2022-03-29T17:51:29.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,857 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 112.553115845s 2022-03-29T17:51:29.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,857 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,857 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.485252380s ub 112.487594604s int 16.000000000s))): submitting ctx 2022-03-29T17:51:29.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,857 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,857 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 104.485252380s ub 112.487594604s int 16.000000000s))): complete 2022-03-29T17:51:29.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,878 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:29.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,878 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:29.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,878 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:51:29.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,878 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 104.488250732s 2022-03-29T17:51:29.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,878 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 104.488250732s (sent 112.487380981s) 2022-03-29T17:51:29.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,878 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,878 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:29.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,878 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,879 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:29.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,889 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:51:29.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,889 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541427612s))): start 2022-03-29T17:51:29.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,889 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541427612s))): got map 13 2022-03-29T17:51:29.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,889 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541427612s))): pg present 2022-03-29T17:51:29.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,889 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541427612s) for pg: 2.5 2022-03-29T17:51:29.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 112.541427612s 2022-03-29T17:51:29.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.541427612s (sent 112.541427612s) 2022-03-29T17:51:29.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541427612s))): submitting ctx 2022-03-29T17:51:29.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:51:29.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541984558s))): start 2022-03-29T17:51:29.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541427612s))): complete 2022-03-29T17:51:29.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541984558s))): got map 13 2022-03-29T17:51:29.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541984558s))): pg present 2022-03-29T17:51:29.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541984558s) for pg: 2.6 2022-03-29T17:51:29.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,890 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 112.541984558s 2022-03-29T17:51:29.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.541984558s (sent 112.541984558s) 2022-03-29T17:51:29.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541984558s))): submitting ctx 2022-03-29T17:51:29.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:51:29.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542152405s))): start 2022-03-29T17:51:29.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542152405s))): got map 13 2022-03-29T17:51:29.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.541984558s))): complete 2022-03-29T17:51:29.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542152405s))): pg present 2022-03-29T17:51:29.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:51:29.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,891 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542312622s))): start 2022-03-29T17:51:29.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542152405s) for pg: 2.7 2022-03-29T17:51:29.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 112.542152405s 2022-03-29T17:51:29.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.542152405s (sent 112.542152405s) 2022-03-29T17:51:29.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542152405s))): submitting ctx 2022-03-29T17:51:29.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542312622s))): got map 13 2022-03-29T17:51:29.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542312622s))): pg present 2022-03-29T17:51:29.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542312622s) for pg: 2.4 2022-03-29T17:51:29.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 112.542312622s 2022-03-29T17:51:29.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,892 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.542312622s (sent 112.542312622s) 2022-03-29T17:51:29.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542312622s))): submitting ctx 2022-03-29T17:51:29.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542152405s))): complete 2022-03-29T17:51:29.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:51:29.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 104.488250732s ub 112.487380981s int 16.000000000s))): start 2022-03-29T17:51:29.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 104.488250732s ub 112.487380981s int 16.000000000s))): got map 13 2022-03-29T17:51:29.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 104.488250732s ub 112.487380981s int 16.000000000s))): pg present 2022-03-29T17:51:29.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 112.542312622s))): complete 2022-03-29T17:51:29.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 104.488250732s ub 112.487380981s int 16.000000000s) for pg: 2.2 2022-03-29T17:51:29.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 104.488250732s ub 112.487380981s int 16.000000000s) 2022-03-29T17:51:29.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 112.589576721s 2022-03-29T17:51:29.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 112.589576721s 2022-03-29T17:51:29.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 104.488250732s ub 112.487380981s int 16.000000000s))): submitting ctx 2022-03-29T17:51:29.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,893 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:29,894 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 104.488250732s ub 112.487380981s int 16.000000000s))): complete 2022-03-29T17:51:29.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:51:29.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487594604s))): start 2022-03-29T17:51:29.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487594604s))): got map 13 2022-03-29T17:51:29.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487594604s))): pg present 2022-03-29T17:51:29.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487594604s) for pg: 2.1 2022-03-29T17:51:29.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 112.487594604s 2022-03-29T17:51:29.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.487594604s (sent 112.487594604s) 2022-03-29T17:51:29.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487594604s))): submitting ctx 2022-03-29T17:51:29.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,897 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:29,898 [shard 0] osd - peering_event(id=76, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487594604s))): complete 2022-03-29T17:51:29.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,930 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:51:29.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,930 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487380981s))): start 2022-03-29T17:51:29.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,930 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487380981s))): got map 13 2022-03-29T17:51:29.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,931 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487380981s))): pg present 2022-03-29T17:51:29.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,931 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487380981s) for pg: 2.2 2022-03-29T17:51:29.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,931 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 112.487380981s 2022-03-29T17:51:29.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,931 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.487380981s (sent 112.487380981s) 2022-03-29T17:51:29.935 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,931 [shard 0] osd - log is not dirty 2022-03-29T17:51:29.935 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,931 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487380981s))): submitting ctx 2022-03-29T17:51:29.935 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,931 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:29.935 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:29,932 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 112.487380981s))): complete 2022-03-29T17:51:30.703 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:30,702 [shard 0] alienstore - stat 2022-03-29T17:51:30.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:30,718 [shard 0] alienstore - stat 2022-03-29T17:51:30.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:30,720 [shard 0] alienstore - stat 2022-03-29T17:51:31.712 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:31,706 [shard 0] alienstore - stat 2022-03-29T17:51:31.726 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:31,722 [shard 0] alienstore - stat 2022-03-29T17:51:31.727 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:31,724 [shard 0] alienstore - stat 2022-03-29T17:51:32.711 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:32,710 [shard 0] alienstore - stat 2022-03-29T17:51:32.728 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:32,726 [shard 0] alienstore - stat 2022-03-29T17:51:32.730 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:32,728 [shard 0] alienstore - stat 2022-03-29T17:51:33.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:33,714 [shard 0] alienstore - stat 2022-03-29T17:51:33.732 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:33,730 [shard 0] alienstore - stat 2022-03-29T17:51:33.734 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:33,732 [shard 0] alienstore - stat 2022-03-29T17:51:34.718 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:34,716 [shard 0] alienstore - stat 2022-03-29T17:51:34.735 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:34,734 [shard 0] alienstore - stat 2022-03-29T17:51:34.738 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:34,737 [shard 0] alienstore - stat 2022-03-29T17:51:34.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,817 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:51:34.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,817 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:51:34.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,817 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:51:34.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,817 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 109.505271912s 2022-03-29T17:51:34.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,817 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 109.505271912s (sent 117.513389587s) 2022-03-29T17:51:34.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,817 [shard 0] osd - log is not dirty 2022-03-29T17:51:34.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,817 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:51:34.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,817 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:34.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,818 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:51:34.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,818 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 109.505271912s ub 117.513389587s int 16.000000000s))): start 2022-03-29T17:51:34.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,818 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 109.505271912s ub 117.513389587s int 16.000000000s))): got map 13 2022-03-29T17:51:34.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,818 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 109.505271912s ub 117.513389587s int 16.000000000s))): pg present 2022-03-29T17:51:34.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,818 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 109.505271912s ub 117.513389587s int 16.000000000s) for pg: 1.0 2022-03-29T17:51:34.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,819 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 109.505271912s ub 117.513389587s int 16.000000000s) 2022-03-29T17:51:34.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,819 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 117.427894592s 2022-03-29T17:51:34.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,819 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 117.427894592s 2022-03-29T17:51:34.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,819 [shard 0] osd - log is not dirty 2022-03-29T17:51:34.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,819 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 109.505271912s ub 117.513389587s int 16.000000000s))): submitting ctx 2022-03-29T17:51:34.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,819 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:34.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,819 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:51:34.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,819 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:51:34.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 117.513389587s))): start 2022-03-29T17:51:34.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 109.505271912s ub 117.513389587s int 16.000000000s))): complete 2022-03-29T17:51:34.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 117.513389587s))): got map 13 2022-03-29T17:51:34.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 117.513389587s))): pg present 2022-03-29T17:51:34.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 117.513389587s) for pg: 1.0 2022-03-29T17:51:34.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 117.513389587s 2022-03-29T17:51:34.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 117.513389587s (sent 117.513389587s) 2022-03-29T17:51:34.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - log is not dirty 2022-03-29T17:51:34.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 117.513389587s))): submitting ctx 2022-03-29T17:51:34.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,820 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:34.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:34,821 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 117.513389587s))): complete 2022-03-29T17:51:35.722 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:35,720 [shard 0] alienstore - stat 2022-03-29T17:51:35.739 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:35,738 [shard 0] alienstore - stat 2022-03-29T17:51:35.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:35,741 [shard 0] alienstore - stat 2022-03-29T17:51:36.727 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:36,725 [shard 0] alienstore - stat 2022-03-29T17:51:36.744 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:36,742 [shard 0] alienstore - stat 2022-03-29T17:51:36.749 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:36,748 [shard 0] alienstore - stat 2022-03-29T17:51:37.729 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:37,728 [shard 0] alienstore - stat 2022-03-29T17:51:37.746 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:37,745 [shard 0] alienstore - stat 2022-03-29T17:51:37.752 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:37,750 [shard 0] alienstore - stat 2022-03-29T17:51:37.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,830 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:37.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,830 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:37.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,831 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:51:37.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,831 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 112.520591736s 2022-03-29T17:51:37.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,831 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.520591736s (sent 120.527061462s) 2022-03-29T17:51:37.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,831 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,831 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:37.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,831 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.840 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,832 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:51:37.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,832 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.520591736s ub 120.527061462s int 16.000000000s))): start 2022-03-29T17:51:37.841 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,832 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.520591736s ub 120.527061462s int 16.000000000s))): got map 13 2022-03-29T17:51:37.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,832 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.520591736s ub 120.527061462s int 16.000000000s))): pg present 2022-03-29T17:51:37.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,832 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.520591736s ub 120.527061462s int 16.000000000s) for pg: 2.3 2022-03-29T17:51:37.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,832 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 112.520591736s ub 120.527061462s int 16.000000000s) 2022-03-29T17:51:37.842 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,832 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 120.466003418s 2022-03-29T17:51:37.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,833 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 120.466003418s 2022-03-29T17:51:37.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,833 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,833 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.520591736s ub 120.527061462s int 16.000000000s))): submitting ctx 2022-03-29T17:51:37.843 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,833 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,833 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:37.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,834 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:51:37.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,834 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 120.527061462s))): start 2022-03-29T17:51:37.845 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,834 [shard 0] osd - peering_event(id=77, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.520591736s ub 120.527061462s int 16.000000000s))): complete 2022-03-29T17:51:37.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,835 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 120.527061462s))): got map 13 2022-03-29T17:51:37.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,835 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 120.527061462s))): pg present 2022-03-29T17:51:37.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,835 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 120.527061462s) for pg: 2.3 2022-03-29T17:51:37.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,835 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 120.527061462s 2022-03-29T17:51:37.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,835 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.527061462s (sent 120.527061462s) 2022-03-29T17:51:37.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,835 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,835 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 120.527061462s))): submitting ctx 2022-03-29T17:51:37.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,835 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,836 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 120.527061462s))): complete 2022-03-29T17:51:37.849 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,845 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:37.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,845 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:37.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,846 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:51:37.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,846 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 112.477554321s 2022-03-29T17:51:37.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,846 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.477554321s (sent 120.479110718s) 2022-03-29T17:51:37.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,846 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,846 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:37.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,846 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,847 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:51:37.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,847 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.477554321s ub 120.479110718s int 16.000000000s))): start 2022-03-29T17:51:37.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,847 [shard 0] osd - peering_event(id=78, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:37.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.477554321s ub 120.479110718s int 16.000000000s))): got map 13 2022-03-29T17:51:37.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.477554321s ub 120.479110718s int 16.000000000s))): pg present 2022-03-29T17:51:37.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.477554321s ub 120.479110718s int 16.000000000s) for pg: 2.0 2022-03-29T17:51:37.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 112.477554321s ub 120.479110718s int 16.000000000s) 2022-03-29T17:51:37.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 120.544578552s 2022-03-29T17:51:37.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 120.544578552s 2022-03-29T17:51:37.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.477554321s ub 120.479110718s int 16.000000000s))): submitting ctx 2022-03-29T17:51:37.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,848 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,849 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.477554321s ub 120.479110718s int 16.000000000s))): complete 2022-03-29T17:51:37.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,850 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:37.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,850 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:37.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,850 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:37.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,850 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:37.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,850 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:37.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,850 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:37.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,851 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:37.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,851 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:37.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,851 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:51:37.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,851 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 112.541984558s 2022-03-29T17:51:37.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,851 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.541984558s (sent 120.547241211s) 2022-03-29T17:51:37.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,851 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,851 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:37.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:51:37.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541984558s ub 120.547241211s int 16.000000000s))): start 2022-03-29T17:51:37.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541984558s ub 120.547241211s int 16.000000000s))): got map 13 2022-03-29T17:51:37.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541984558s ub 120.547241211s int 16.000000000s))): pg present 2022-03-29T17:51:37.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541984558s ub 120.547241211s int 16.000000000s) for pg: 2.6 2022-03-29T17:51:37.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 112.541984558s ub 120.547241211s int 16.000000000s) 2022-03-29T17:51:37.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 120.461639404s 2022-03-29T17:51:37.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 120.461639404s 2022-03-29T17:51:37.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541984558s ub 120.547241211s int 16.000000000s))): submitting ctx 2022-03-29T17:51:37.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,852 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,853 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:51:37.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,853 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 112.542312622s 2022-03-29T17:51:37.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,853 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.542312622s (sent 120.549385071s) 2022-03-29T17:51:37.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,853 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,853 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:37.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,853 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,853 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541984558s ub 120.547241211s int 16.000000000s))): complete 2022-03-29T17:51:37.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:51:37.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 112.541427612s 2022-03-29T17:51:37.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.541427612s (sent 120.550132751s) 2022-03-29T17:51:37.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:37.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:51:37.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 112.542152405s 2022-03-29T17:51:37.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.542152405s (sent 120.550598145s) 2022-03-29T17:51:37.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:37.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,854 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,855 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:51:37.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,855 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542312622s ub 120.549385071s int 16.000000000s))): start 2022-03-29T17:51:37.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,855 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542312622s ub 120.549385071s int 16.000000000s))): got map 13 2022-03-29T17:51:37.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542312622s ub 120.549385071s int 16.000000000s))): pg present 2022-03-29T17:51:37.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:51:37.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541427612s ub 120.550132751s int 16.000000000s))): start 2022-03-29T17:51:37.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542312622s ub 120.549385071s int 16.000000000s) for pg: 2.4 2022-03-29T17:51:37.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 112.542312622s ub 120.549385071s int 16.000000000s) 2022-03-29T17:51:37.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 120.465484619s 2022-03-29T17:51:37.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 120.465484619s 2022-03-29T17:51:37.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542312622s ub 120.549385071s int 16.000000000s))): submitting ctx 2022-03-29T17:51:37.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,856 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541427612s ub 120.550132751s int 16.000000000s))): got map 13 2022-03-29T17:51:37.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541427612s ub 120.550132751s int 16.000000000s))): pg present 2022-03-29T17:51:37.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541427612s ub 120.550132751s int 16.000000000s) for pg: 2.5 2022-03-29T17:51:37.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 112.541427612s ub 120.550132751s int 16.000000000s) 2022-03-29T17:51:37.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 120.466384888s 2022-03-29T17:51:37.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 120.466384888s 2022-03-29T17:51:37.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541427612s ub 120.550132751s int 16.000000000s))): submitting ctx 2022-03-29T17:51:37.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:51:37.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,857 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542152405s ub 120.550598145s int 16.000000000s))): start 2022-03-29T17:51:37.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542312622s ub 120.549385071s int 16.000000000s))): complete 2022-03-29T17:51:37.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:37.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:37.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:51:37.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 112.487594604s 2022-03-29T17:51:37.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.487594604s (sent 120.491569519s) 2022-03-29T17:51:37.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542152405s ub 120.550598145s int 16.000000000s))): got map 13 2022-03-29T17:51:37.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.541427612s ub 120.550132751s int 16.000000000s))): complete 2022-03-29T17:51:37.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,859 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542152405s ub 120.550598145s int 16.000000000s))): pg present 2022-03-29T17:51:37.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,859 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542152405s ub 120.550598145s int 16.000000000s) for pg: 2.7 2022-03-29T17:51:37.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,858 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:37.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,859 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,859 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 112.542152405s ub 120.550598145s int 16.000000000s) 2022-03-29T17:51:37.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,859 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 120.468673706s 2022-03-29T17:51:37.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,859 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 120.468673706s 2022-03-29T17:51:37.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,859 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,859 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542152405s ub 120.550598145s int 16.000000000s))): submitting ctx 2022-03-29T17:51:37.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,860 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,860 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:51:37.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,860 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.479110718s))): start 2022-03-29T17:51:37.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,860 [shard 0] osd - peering_event(id=79, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:37.889 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,860 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 112.542152405s ub 120.550598145s int 16.000000000s))): complete 2022-03-29T17:51:37.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,861 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.479110718s))): got map 13 2022-03-29T17:51:37.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,861 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.479110718s))): pg present 2022-03-29T17:51:37.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,861 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.479110718s) for pg: 2.0 2022-03-29T17:51:37.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,861 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 120.479110718s 2022-03-29T17:51:37.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,861 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.479110718s (sent 120.479110718s) 2022-03-29T17:51:37.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,861 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,861 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.479110718s))): submitting ctx 2022-03-29T17:51:37.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,861 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,862 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:37.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,862 [shard 0] osd - peering_event(id=80, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.479110718s))): complete 2022-03-29T17:51:37.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,862 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:37.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,862 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:37.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,862 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:37.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,863 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:51:37.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,863 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.487594604s ub 120.491569519s int 16.000000000s))): start 2022-03-29T17:51:37.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,863 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.487594604s ub 120.491569519s int 16.000000000s))): got map 13 2022-03-29T17:51:37.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,863 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.487594604s ub 120.491569519s int 16.000000000s))): pg present 2022-03-29T17:51:37.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,864 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:51:37.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,864 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.547241211s))): start 2022-03-29T17:51:37.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,864 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.487594604s ub 120.491569519s int 16.000000000s) for pg: 2.1 2022-03-29T17:51:37.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,864 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 112.487594604s ub 120.491569519s int 16.000000000s) 2022-03-29T17:51:37.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,865 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 120.560623169s 2022-03-29T17:51:37.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,865 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 120.560623169s 2022-03-29T17:51:37.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,865 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,865 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.487594604s ub 120.491569519s int 16.000000000s))): submitting ctx 2022-03-29T17:51:37.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,865 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,865 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.547241211s))): got map 13 2022-03-29T17:51:37.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,865 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.547241211s))): pg present 2022-03-29T17:51:37.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,865 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 112.487594604s ub 120.491569519s int 16.000000000s))): complete 2022-03-29T17:51:37.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,866 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.547241211s) for pg: 2.6 2022-03-29T17:51:37.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,866 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 120.547241211s 2022-03-29T17:51:37.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,866 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.547241211s (sent 120.547241211s) 2022-03-29T17:51:37.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,866 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,866 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.547241211s))): submitting ctx 2022-03-29T17:51:37.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,866 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,866 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:51:37.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,866 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.549385071s))): start 2022-03-29T17:51:37.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,867 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.547241211s))): complete 2022-03-29T17:51:37.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,867 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.549385071s))): got map 13 2022-03-29T17:51:37.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,867 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.549385071s))): pg present 2022-03-29T17:51:37.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,867 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:51:37.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550132751s))): start 2022-03-29T17:51:37.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.549385071s) for pg: 2.4 2022-03-29T17:51:37.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 120.549385071s 2022-03-29T17:51:37.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.549385071s (sent 120.549385071s) 2022-03-29T17:51:37.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.549385071s))): submitting ctx 2022-03-29T17:51:37.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550132751s))): got map 13 2022-03-29T17:51:37.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,868 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550132751s))): pg present 2022-03-29T17:51:37.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550132751s) for pg: 2.5 2022-03-29T17:51:37.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 120.550132751s 2022-03-29T17:51:37.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.550132751s (sent 120.550132751s) 2022-03-29T17:51:37.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550132751s))): submitting ctx 2022-03-29T17:51:37.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.549385071s))): complete 2022-03-29T17:51:37.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:51:37.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550598145s))): start 2022-03-29T17:51:37.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,869 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550132751s))): complete 2022-03-29T17:51:37.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550598145s))): got map 13 2022-03-29T17:51:37.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550598145s))): pg present 2022-03-29T17:51:37.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550598145s) for pg: 2.7 2022-03-29T17:51:37.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 120.550598145s 2022-03-29T17:51:37.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.550598145s (sent 120.550598145s) 2022-03-29T17:51:37.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550598145s))): submitting ctx 2022-03-29T17:51:37.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,870 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 120.550598145s))): complete 2022-03-29T17:51:37.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,878 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:37.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,878 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:37.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,878 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:51:37.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,878 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 112.487380981s 2022-03-29T17:51:37.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,878 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 112.487380981s (sent 120.487480164s) 2022-03-29T17:51:37.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,878 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,878 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:37.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,878 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,879 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:37.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,880 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:51:37.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,880 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 112.487380981s ub 120.487480164s int 16.000000000s))): start 2022-03-29T17:51:37.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,880 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 112.487380981s ub 120.487480164s int 16.000000000s))): got map 13 2022-03-29T17:51:37.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,880 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 112.487380981s ub 120.487480164s int 16.000000000s))): pg present 2022-03-29T17:51:37.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,880 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 112.487380981s ub 120.487480164s int 16.000000000s) for pg: 2.2 2022-03-29T17:51:37.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,880 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 112.487380981s ub 120.487480164s int 16.000000000s) 2022-03-29T17:51:37.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,880 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 120.576751709s 2022-03-29T17:51:37.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,880 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 120.576751709s 2022-03-29T17:51:37.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,881 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,881 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 112.487380981s ub 120.487480164s int 16.000000000s))): submitting ctx 2022-03-29T17:51:37.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,881 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:37,881 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 112.487380981s ub 120.487480164s int 16.000000000s))): complete 2022-03-29T17:51:37.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,904 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:51:37.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,904 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.491569519s))): start 2022-03-29T17:51:37.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,905 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.491569519s))): got map 13 2022-03-29T17:51:37.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,905 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.491569519s))): pg present 2022-03-29T17:51:37.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,905 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.491569519s) for pg: 2.1 2022-03-29T17:51:37.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,905 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 120.491569519s 2022-03-29T17:51:37.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,905 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.491569519s (sent 120.491569519s) 2022-03-29T17:51:37.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,905 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,905 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.491569519s))): submitting ctx 2022-03-29T17:51:37.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,905 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:37,906 [shard 0] osd - peering_event(id=81, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.491569519s))): complete 2022-03-29T17:51:37.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,919 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:51:37.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,919 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.487480164s))): start 2022-03-29T17:51:37.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,920 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.487480164s))): got map 13 2022-03-29T17:51:37.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,920 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.487480164s))): pg present 2022-03-29T17:51:37.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,920 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.487480164s) for pg: 2.2 2022-03-29T17:51:37.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,920 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 120.487480164s 2022-03-29T17:51:37.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,920 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.487480164s (sent 120.487480164s) 2022-03-29T17:51:37.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,920 [shard 0] osd - log is not dirty 2022-03-29T17:51:37.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,920 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.487480164s))): submitting ctx 2022-03-29T17:51:37.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,920 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:37.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:37,921 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 120.487480164s))): complete 2022-03-29T17:51:38.731 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:38,729 [shard 0] alienstore - stat 2022-03-29T17:51:38.754 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:38,752 [shard 0] alienstore - stat 2022-03-29T17:51:38.755 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:38,753 [shard 0] alienstore - stat 2022-03-29T17:51:39.732 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:39,731 [shard 0] alienstore - stat 2022-03-29T17:51:39.755 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:39,753 [shard 0] alienstore - stat 2022-03-29T17:51:39.760 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:39,759 [shard 0] alienstore - stat 2022-03-29T17:51:40.741 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:40,739 [shard 0] alienstore - stat 2022-03-29T17:51:40.761 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:40,760 [shard 0] alienstore - stat 2022-03-29T17:51:40.761 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:40,760 [shard 0] alienstore - stat 2022-03-29T17:51:41.742 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:41,741 [shard 0] alienstore - stat 2022-03-29T17:51:41.763 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:41,761 [shard 0] alienstore - stat 2022-03-29T17:51:41.763 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:41,762 [shard 0] alienstore - stat 2022-03-29T17:51:42.744 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:42,742 [shard 0] alienstore - stat 2022-03-29T17:51:42.764 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:42,763 [shard 0] alienstore - stat 2022-03-29T17:51:42.765 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:42,763 [shard 0] alienstore - stat 2022-03-29T17:51:42.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,822 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:51:42.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,823 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:51:42.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,823 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:51:42.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,823 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 117.513389587s 2022-03-29T17:51:42.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,823 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 117.513389587s (sent 125.519165039s) 2022-03-29T17:51:42.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,823 [shard 0] osd - log is not dirty 2022-03-29T17:51:42.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,823 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:51:42.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,823 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:42.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,824 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:51:42.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,824 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 117.513389587s ub 125.519165039s int 16.000000000s))): start 2022-03-29T17:51:42.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,825 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:51:42.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,825 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 117.513389587s ub 125.519165039s int 16.000000000s))): got map 13 2022-03-29T17:51:42.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,825 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 117.513389587s ub 125.519165039s int 16.000000000s))): pg present 2022-03-29T17:51:42.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,825 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 117.513389587s ub 125.519165039s int 16.000000000s) for pg: 1.0 2022-03-29T17:51:42.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,826 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 117.513389587s ub 125.519165039s int 16.000000000s) 2022-03-29T17:51:42.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,826 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 125.435234070s 2022-03-29T17:51:42.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,826 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 125.435234070s 2022-03-29T17:51:42.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,826 [shard 0] osd - log is not dirty 2022-03-29T17:51:42.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,826 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 117.513389587s ub 125.519165039s int 16.000000000s))): submitting ctx 2022-03-29T17:51:42.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,826 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:42.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:51:42.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 125.519165039s))): start 2022-03-29T17:51:42.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 125.519165039s))): got map 13 2022-03-29T17:51:42.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 125.519165039s))): pg present 2022-03-29T17:51:42.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 125.519165039s) for pg: 1.0 2022-03-29T17:51:42.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 125.519165039s 2022-03-29T17:51:42.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 125.519165039s (sent 125.519165039s) 2022-03-29T17:51:42.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - log is not dirty 2022-03-29T17:51:42.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 125.519165039s))): submitting ctx 2022-03-29T17:51:42.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,827 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:42.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:42,828 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 117.513389587s ub 125.519165039s int 16.000000000s))): complete 2022-03-29T17:51:42.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:42,828 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 125.519165039s))): complete 2022-03-29T17:51:43.745 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:43,743 [shard 0] alienstore - stat 2022-03-29T17:51:43.765 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:43,764 [shard 0] alienstore - stat 2022-03-29T17:51:43.766 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:43,765 [shard 0] alienstore - stat 2022-03-29T17:51:44.751 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:44,749 [shard 0] alienstore - stat 2022-03-29T17:51:44.770 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:44,769 [shard 0] alienstore - stat 2022-03-29T17:51:44.771 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:44,770 [shard 0] alienstore - stat 2022-03-29T17:51:45.759 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:45,757 [shard 0] alienstore - stat 2022-03-29T17:51:45.779 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:45,778 [shard 0] alienstore - stat 2022-03-29T17:51:45.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:45,778 [shard 0] alienstore - stat 2022-03-29T17:51:45.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,837 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:45.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,838 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:45.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,838 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:51:45.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,838 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 120.527061462s 2022-03-29T17:51:45.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,838 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.527061462s (sent 128.534225464s) 2022-03-29T17:51:45.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,838 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,838 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:45.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,838 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,839 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:51:45.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,839 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.527061462s ub 128.534225464s int 16.000000000s))): start 2022-03-29T17:51:45.846 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.527061462s ub 128.534225464s int 16.000000000s))): got map 13 2022-03-29T17:51:45.847 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.527061462s ub 128.534225464s int 16.000000000s))): pg present 2022-03-29T17:51:45.847 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.527061462s ub 128.534225464s int 16.000000000s) for pg: 2.3 2022-03-29T17:51:45.847 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 120.527061462s ub 128.534225464s int 16.000000000s) 2022-03-29T17:51:45.847 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 128.473602295s 2022-03-29T17:51:45.848 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 128.473602295s 2022-03-29T17:51:45.848 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.848 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.527061462s ub 128.534225464s int 16.000000000s))): submitting ctx 2022-03-29T17:51:45.848 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,840 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,841 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:45.849 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,841 [shard 0] osd - peering_event(id=82, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.527061462s ub 128.534225464s int 16.000000000s))): complete 2022-03-29T17:51:45.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,842 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:51:45.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,842 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 128.534225464s))): start 2022-03-29T17:51:45.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,842 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 128.534225464s))): got map 13 2022-03-29T17:51:45.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,842 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 128.534225464s))): pg present 2022-03-29T17:51:45.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,843 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 128.534225464s) for pg: 2.3 2022-03-29T17:51:45.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,843 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 128.534225464s 2022-03-29T17:51:45.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,843 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.534225464s (sent 128.534225464s) 2022-03-29T17:51:45.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,843 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,843 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 128.534225464s))): submitting ctx 2022-03-29T17:51:45.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,843 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,843 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 128.534225464s))): complete 2022-03-29T17:51:45.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,848 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:45.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,848 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:45.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,848 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:51:45.855 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,848 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 120.479110718s 2022-03-29T17:51:45.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,848 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.479110718s (sent 128.481719971s) 2022-03-29T17:51:45.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,848 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,848 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:45.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,848 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,849 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:51:45.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,849 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.479110718s ub 128.481719971s int 16.000000000s))): start 2022-03-29T17:51:45.858 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - peering_event(id=83, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:45.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.479110718s ub 128.481719971s int 16.000000000s))): got map 13 2022-03-29T17:51:45.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.479110718s ub 128.481719971s int 16.000000000s))): pg present 2022-03-29T17:51:45.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.479110718s ub 128.481719971s int 16.000000000s) for pg: 2.0 2022-03-29T17:51:45.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 120.479110718s ub 128.481719971s int 16.000000000s) 2022-03-29T17:51:45.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 128.546707153s 2022-03-29T17:51:45.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 128.546707153s 2022-03-29T17:51:45.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,850 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.479110718s ub 128.481719971s int 16.000000000s))): submitting ctx 2022-03-29T17:51:45.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,851 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.479110718s ub 128.481719971s int 16.000000000s))): complete 2022-03-29T17:51:45.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,857 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:45.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,857 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:45.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:45.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:45.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:45.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:45.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:45.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:45.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:51:45.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 120.550598145s 2022-03-29T17:51:45.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.550598145s (sent 128.554473877s) 2022-03-29T17:51:45.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:45.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,858 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:45.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:45.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:51:45.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 120.491569519s 2022-03-29T17:51:45.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.491569519s (sent 128.492202759s) 2022-03-29T17:51:45.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:45.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:51:45.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,859 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.481719971s))): start 2022-03-29T17:51:45.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - peering_event(id=84, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:45.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.481719971s))): got map 13 2022-03-29T17:51:45.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.481719971s))): pg present 2022-03-29T17:51:45.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.481719971s) for pg: 2.0 2022-03-29T17:51:45.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 128.481719971s 2022-03-29T17:51:45.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.481719971s (sent 128.481719971s) 2022-03-29T17:51:45.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.481719971s))): submitting ctx 2022-03-29T17:51:45.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,860 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,861 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:51:45.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,861 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 120.550132751s 2022-03-29T17:51:45.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,861 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.550132751s (sent 128.557189941s) 2022-03-29T17:51:45.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,861 [shard 0] osd - peering_event(id=85, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.481719971s))): complete 2022-03-29T17:51:45.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,861 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,861 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:45.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,861 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,862 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:51:45.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,862 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 120.547241211s 2022-03-29T17:51:45.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,862 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:51:45.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,862 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550598145s ub 128.554473877s int 16.000000000s))): start 2022-03-29T17:51:45.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550598145s ub 128.554473877s int 16.000000000s))): got map 13 2022-03-29T17:51:45.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550598145s ub 128.554473877s int 16.000000000s))): pg present 2022-03-29T17:51:45.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:51:45.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550132751s ub 128.557189941s int 16.000000000s))): start 2022-03-29T17:51:45.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550598145s ub 128.554473877s int 16.000000000s) for pg: 2.7 2022-03-29T17:51:45.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 120.550598145s ub 128.554473877s int 16.000000000s) 2022-03-29T17:51:45.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 128.472732544s 2022-03-29T17:51:45.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 128.472732544s 2022-03-29T17:51:45.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,863 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550598145s ub 128.554473877s int 16.000000000s))): submitting ctx 2022-03-29T17:51:45.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550132751s ub 128.557189941s int 16.000000000s))): got map 13 2022-03-29T17:51:45.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550132751s ub 128.557189941s int 16.000000000s))): pg present 2022-03-29T17:51:45.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550132751s ub 128.557189941s int 16.000000000s) for pg: 2.5 2022-03-29T17:51:45.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 120.550132751s ub 128.557189941s int 16.000000000s) 2022-03-29T17:51:45.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 128.473556519s 2022-03-29T17:51:45.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 128.473556519s 2022-03-29T17:51:45.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550132751s ub 128.557189941s int 16.000000000s))): submitting ctx 2022-03-29T17:51:45.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,864 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,865 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550598145s ub 128.554473877s int 16.000000000s))): complete 2022-03-29T17:51:45.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,865 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.550132751s ub 128.557189941s int 16.000000000s))): complete 2022-03-29T17:51:45.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,865 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.547241211s (sent 128.558242798s) 2022-03-29T17:51:45.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,865 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,865 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:45.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,865 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,866 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:51:45.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,866 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 120.549385071s 2022-03-29T17:51:45.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,866 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:51:45.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,866 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.547241211s ub 128.558242798s int 16.000000000s))): start 2022-03-29T17:51:45.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.547241211s ub 128.558242798s int 16.000000000s))): got map 13 2022-03-29T17:51:45.889 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.547241211s ub 128.558242798s int 16.000000000s))): pg present 2022-03-29T17:51:45.889 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.547241211s ub 128.558242798s int 16.000000000s) for pg: 2.6 2022-03-29T17:51:45.889 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 120.547241211s ub 128.558242798s int 16.000000000s) 2022-03-29T17:51:45.889 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 128.476181030s 2022-03-29T17:51:45.889 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 128.476181030s 2022-03-29T17:51:45.890 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.890 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.547241211s ub 128.558242798s int 16.000000000s))): submitting ctx 2022-03-29T17:51:45.890 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,867 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,866 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.549385071s (sent 128.562225342s) 2022-03-29T17:51:45.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,868 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,868 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:45.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,868 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,868 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:45.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,868 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.547241211s ub 128.558242798s int 16.000000000s))): complete 2022-03-29T17:51:45.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:51:45.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.549385071s ub 128.562225342s int 16.000000000s))): start 2022-03-29T17:51:45.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.549385071s ub 128.562225342s int 16.000000000s))): got map 13 2022-03-29T17:51:45.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.549385071s ub 128.562225342s int 16.000000000s))): pg present 2022-03-29T17:51:45.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.549385071s ub 128.562225342s int 16.000000000s) for pg: 2.4 2022-03-29T17:51:45.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 120.549385071s ub 128.562225342s int 16.000000000s) 2022-03-29T17:51:45.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 128.478668213s 2022-03-29T17:51:45.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 128.478668213s 2022-03-29T17:51:45.896 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,869 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.896 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,870 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.549385071s ub 128.562225342s int 16.000000000s))): submitting ctx 2022-03-29T17:51:45.896 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,870 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,870 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:45.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,870 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:45.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,870 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:51:45.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,870 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 120.549385071s ub 128.562225342s int 16.000000000s))): complete 2022-03-29T17:51:45.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,871 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.491569519s ub 128.492202759s int 16.000000000s))): start 2022-03-29T17:51:45.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,871 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.491569519s ub 128.492202759s int 16.000000000s))): got map 13 2022-03-29T17:51:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,871 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,871 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.491569519s ub 128.492202759s int 16.000000000s))): pg present 2022-03-29T17:51:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,872 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.491569519s ub 128.492202759s int 16.000000000s) for pg: 2.1 2022-03-29T17:51:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,872 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 120.491569519s ub 128.492202759s int 16.000000000s) 2022-03-29T17:51:45.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,872 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 128.568695068s 2022-03-29T17:51:45.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,872 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 128.568695068s 2022-03-29T17:51:45.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,872 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,872 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.491569519s ub 128.492202759s int 16.000000000s))): submitting ctx 2022-03-29T17:51:45.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,872 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,873 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:51:45.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,873 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.554473877s))): start 2022-03-29T17:51:45.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,873 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 120.491569519s ub 128.492202759s int 16.000000000s))): complete 2022-03-29T17:51:45.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,873 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.554473877s))): got map 13 2022-03-29T17:51:45.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.554473877s))): pg present 2022-03-29T17:51:45.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.554473877s) for pg: 2.7 2022-03-29T17:51:45.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 128.554473877s 2022-03-29T17:51:45.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.554473877s (sent 128.554473877s) 2022-03-29T17:51:45.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.554473877s))): submitting ctx 2022-03-29T17:51:45.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:51:45.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,874 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.557189941s))): start 2022-03-29T17:51:45.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,875 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.557189941s))): got map 13 2022-03-29T17:51:45.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,875 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.557189941s))): pg present 2022-03-29T17:51:45.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,875 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.557189941s) for pg: 2.5 2022-03-29T17:51:45.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 128.557189941s 2022-03-29T17:51:45.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.557189941s (sent 128.557189941s) 2022-03-29T17:51:45.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.557189941s))): submitting ctx 2022-03-29T17:51:45.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.554473877s))): complete 2022-03-29T17:51:45.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:51:45.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.558242798s))): start 2022-03-29T17:51:45.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,876 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.558242798s))): got map 13 2022-03-29T17:51:45.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.558242798s))): pg present 2022-03-29T17:51:45.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.558242798s) for pg: 2.6 2022-03-29T17:51:45.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 128.558242798s 2022-03-29T17:51:45.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.558242798s (sent 128.558242798s) 2022-03-29T17:51:45.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.558242798s))): submitting ctx 2022-03-29T17:51:45.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:51:45.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,877 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.562225342s))): start 2022-03-29T17:51:45.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.562225342s))): got map 13 2022-03-29T17:51:45.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.562225342s))): pg present 2022-03-29T17:51:45.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.562225342s) for pg: 2.4 2022-03-29T17:51:45.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 128.562225342s 2022-03-29T17:51:45.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.562225342s (sent 128.562225342s) 2022-03-29T17:51:45.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.562225342s))): submitting ctx 2022-03-29T17:51:45.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,878 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.557189941s))): complete 2022-03-29T17:51:45.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,881 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.558242798s))): complete 2022-03-29T17:51:45.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,883 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 128.562225342s))): complete 2022-03-29T17:51:45.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,887 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:45.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,888 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:45.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,888 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:51:45.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,888 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 120.487480164s 2022-03-29T17:51:45.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,888 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 120.487480164s (sent 128.497055054s) 2022-03-29T17:51:45.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,888 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,888 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:45.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,888 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,889 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:45.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,889 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:51:45.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,889 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 120.487480164s ub 128.497055054s int 16.000000000s))): start 2022-03-29T17:51:45.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,889 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 120.487480164s ub 128.497055054s int 16.000000000s))): got map 13 2022-03-29T17:51:45.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,889 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 120.487480164s ub 128.497055054s int 16.000000000s))): pg present 2022-03-29T17:51:45.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,890 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 120.487480164s ub 128.497055054s int 16.000000000s) for pg: 2.2 2022-03-29T17:51:45.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,890 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 120.487480164s ub 128.497055054s int 16.000000000s) 2022-03-29T17:51:45.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,890 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 128.586090088s 2022-03-29T17:51:45.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,890 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 128.586090088s 2022-03-29T17:51:45.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,890 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,890 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 120.487480164s ub 128.497055054s int 16.000000000s))): submitting ctx 2022-03-29T17:51:45.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,890 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:45,890 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 120.487480164s ub 128.497055054s int 16.000000000s))): complete 2022-03-29T17:51:45.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,912 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:51:45.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,912 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.492202759s))): start 2022-03-29T17:51:45.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,913 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.492202759s))): got map 13 2022-03-29T17:51:45.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,913 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.492202759s))): pg present 2022-03-29T17:51:45.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,913 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.492202759s) for pg: 2.1 2022-03-29T17:51:45.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,913 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 128.492202759s 2022-03-29T17:51:45.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,913 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.492202759s (sent 128.492202759s) 2022-03-29T17:51:45.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,913 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,913 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.492202759s))): submitting ctx 2022-03-29T17:51:45.927 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,913 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.927 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:45,914 [shard 0] osd - peering_event(id=86, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.492202759s))): complete 2022-03-29T17:51:45.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,929 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:51:45.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,929 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.497055054s))): start 2022-03-29T17:51:45.932 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,930 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.497055054s))): got map 13 2022-03-29T17:51:45.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,930 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.497055054s))): pg present 2022-03-29T17:51:45.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,930 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.497055054s) for pg: 2.2 2022-03-29T17:51:45.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,930 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 128.497055054s 2022-03-29T17:51:45.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,930 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.497055054s (sent 128.497055054s) 2022-03-29T17:51:45.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,930 [shard 0] osd - log is not dirty 2022-03-29T17:51:45.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,930 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.497055054s))): submitting ctx 2022-03-29T17:51:45.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,930 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:45.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:45,931 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 128.497055054s))): complete 2022-03-29T17:51:46.761 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:46,759 [shard 0] alienstore - stat 2022-03-29T17:51:46.783 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:46,782 [shard 0] alienstore - stat 2022-03-29T17:51:46.784 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:46,783 [shard 0] alienstore - stat 2022-03-29T17:51:47.762 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:47,761 [shard 0] alienstore - stat 2022-03-29T17:51:47.788 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:47,786 [shard 0] alienstore - stat 2022-03-29T17:51:47.788 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:47,787 [shard 0] alienstore - stat 2022-03-29T17:51:48.764 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:48,762 [shard 0] alienstore - stat 2022-03-29T17:51:48.789 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:48,788 [shard 0] alienstore - stat 2022-03-29T17:51:48.790 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:48,788 [shard 0] alienstore - stat 2022-03-29T17:51:49.765 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:49,763 [shard 0] alienstore - stat 2022-03-29T17:51:49.790 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:49,789 [shard 0] alienstore - stat 2022-03-29T17:51:49.791 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:49,789 [shard 0] alienstore - stat 2022-03-29T17:51:50.766 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:50,764 [shard 0] alienstore - stat 2022-03-29T17:51:50.791 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:50,790 [shard 0] alienstore - stat 2022-03-29T17:51:50.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:50,791 [shard 0] alienstore - stat 2022-03-29T17:51:50.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,830 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:51:50.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,830 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:51:50.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,830 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:51:50.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,830 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 125.519165039s 2022-03-29T17:51:50.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,830 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 125.519165039s (sent 133.526550293s) 2022-03-29T17:51:50.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,831 [shard 0] osd - log is not dirty 2022-03-29T17:51:50.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,831 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:51:50.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,831 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:50.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:51:50.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 125.519165039s ub 133.526550293s int 16.000000000s))): start 2022-03-29T17:51:50.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 125.519165039s ub 133.526550293s int 16.000000000s))): got map 13 2022-03-29T17:51:50.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 125.519165039s ub 133.526550293s int 16.000000000s))): pg present 2022-03-29T17:51:50.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 125.519165039s ub 133.526550293s int 16.000000000s) for pg: 1.0 2022-03-29T17:51:50.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 125.519165039s ub 133.526550293s int 16.000000000s) 2022-03-29T17:51:50.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 133.441635132s 2022-03-29T17:51:50.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 133.441635132s 2022-03-29T17:51:50.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - log is not dirty 2022-03-29T17:51:50.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 125.519165039s ub 133.526550293s int 16.000000000s))): submitting ctx 2022-03-29T17:51:50.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,832 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:50.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,833 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:51:50.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:50,833 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=1 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 125.519165039s ub 133.526550293s int 16.000000000s))): complete 2022-03-29T17:51:50.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,834 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:51:50.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,834 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 133.526550293s))): start 2022-03-29T17:51:50.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,834 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 133.526550293s))): got map 13 2022-03-29T17:51:50.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,834 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 133.526550293s))): pg present 2022-03-29T17:51:50.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,834 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 133.526550293s) for pg: 1.0 2022-03-29T17:51:50.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,834 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 133.526550293s 2022-03-29T17:51:50.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,834 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 133.526550293s (sent 133.526550293s) 2022-03-29T17:51:50.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,835 [shard 0] osd - log is not dirty 2022-03-29T17:51:50.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,835 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 133.526550293s))): submitting ctx 2022-03-29T17:51:50.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,835 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:50.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:50,835 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=0 pgid=1.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 133.526550293s))): complete 2022-03-29T17:51:51.771 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:51,769 [shard 0] alienstore - stat 2022-03-29T17:51:51.794 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:51,791 [shard 0] alienstore - stat 2022-03-29T17:51:51.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:51,792 [shard 0] alienstore - stat 2022-03-29T17:51:52.772 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:52,771 [shard 0] alienstore - stat 2022-03-29T17:51:52.794 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:52,792 [shard 0] alienstore - stat 2022-03-29T17:51:52.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:52,793 [shard 0] alienstore - stat 2022-03-29T17:51:53.774 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:53,772 [shard 0] alienstore - stat 2022-03-29T17:51:53.796 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:53,794 [shard 0] alienstore - stat 2022-03-29T17:51:53.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:53,794 [shard 0] alienstore - stat 2022-03-29T17:51:53.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,844 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:53.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,844 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:53.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,844 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:51:53.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,845 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 128.534225464s 2022-03-29T17:51:53.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,845 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.534225464s (sent 136.540603638s) 2022-03-29T17:51:53.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,845 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,845 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:53.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,845 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.850 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,846 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:51:53.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,846 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.534225464s ub 136.540603638s int 16.000000000s))): start 2022-03-29T17:51:53.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,846 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.534225464s ub 136.540603638s int 16.000000000s))): got map 13 2022-03-29T17:51:53.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,847 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.534225464s ub 136.540603638s int 16.000000000s))): pg present 2022-03-29T17:51:53.851 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,847 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.534225464s ub 136.540603638s int 16.000000000s) for pg: 2.3 2022-03-29T17:51:53.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,847 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 128.534225464s ub 136.540603638s int 16.000000000s) 2022-03-29T17:51:53.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,847 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 136.480316162s 2022-03-29T17:51:53.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,847 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 136.480316162s 2022-03-29T17:51:53.852 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,847 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,847 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.534225464s ub 136.540603638s int 16.000000000s))): submitting ctx 2022-03-29T17:51:53.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,847 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,848 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:53.854 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,848 [shard 0] osd - peering_event(id=87, detail=PeeringEvent(from=1 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.534225464s ub 136.540603638s int 16.000000000s))): complete 2022-03-29T17:51:53.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,848 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:51:53.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,848 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 136.540603638s))): start 2022-03-29T17:51:53.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,849 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 136.540603638s))): got map 13 2022-03-29T17:51:53.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,849 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 136.540603638s))): pg present 2022-03-29T17:51:53.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,849 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 136.540603638s) for pg: 2.3 2022-03-29T17:51:53.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,849 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 136.540603638s 2022-03-29T17:51:53.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,849 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.540603638s (sent 136.540603638s) 2022-03-29T17:51:53.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,849 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,850 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 136.540603638s))): submitting ctx 2022-03-29T17:51:53.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,850 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,850 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=2.3 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.2 pg_lease_ack(ruub 136.540603638s))): complete 2022-03-29T17:51:53.859 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,854 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:53.859 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,854 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:53.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,855 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:51:53.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,855 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 128.481719971s 2022-03-29T17:51:53.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,855 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.481719971s (sent 136.488082886s) 2022-03-29T17:51:53.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,855 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.861 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,855 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:53.861 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,855 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,855 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:51:53.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,856 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.481719971s ub 136.488082886s int 16.000000000s))): start 2022-03-29T17:51:53.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,856 [shard 0] osd - peering_event(id=88, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:53.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,856 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.481719971s ub 136.488082886s int 16.000000000s))): got map 13 2022-03-29T17:51:53.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,856 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.481719971s ub 136.488082886s int 16.000000000s))): pg present 2022-03-29T17:51:53.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,857 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.481719971s ub 136.488082886s int 16.000000000s) for pg: 2.0 2022-03-29T17:51:53.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,857 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 128.481719971s ub 136.488082886s int 16.000000000s) 2022-03-29T17:51:53.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,857 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 136.553009033s 2022-03-29T17:51:53.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,857 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 136.553009033s 2022-03-29T17:51:53.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,857 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,857 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.481719971s ub 136.488082886s int 16.000000000s))): submitting ctx 2022-03-29T17:51:53.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,857 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,857 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=2 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.481719971s ub 136.488082886s int 16.000000000s))): complete 2022-03-29T17:51:53.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:53.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:53.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:53.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:53.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:53.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:53.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:53.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:53.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:53.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,864 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:53.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,865 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:51:53.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,865 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 128.492202759s 2022-03-29T17:51:53.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,865 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.492202759s (sent 136.498077393s) 2022-03-29T17:51:53.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,865 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,865 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:53.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,865 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,865 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:51:53.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,865 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.488082886s))): start 2022-03-29T17:51:53.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - peering_event(id=89, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:53.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.488082886s))): got map 13 2022-03-29T17:51:53.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.488082886s))): pg present 2022-03-29T17:51:53.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.488082886s) for pg: 2.0 2022-03-29T17:51:53.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 136.488082886s 2022-03-29T17:51:53.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.488082886s (sent 136.488082886s) 2022-03-29T17:51:53.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.488082886s))): submitting ctx 2022-03-29T17:51:53.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,866 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - peering_event(id=90, detail=PeeringEvent(from=1 pgid=2.0 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.488082886s))): complete 2022-03-29T17:51:53.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:51:53.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 128.557189941s 2022-03-29T17:51:53.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.557189941s (sent 136.563491821s) 2022-03-29T17:51:53.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:53.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:51:53.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,867 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 128.554473877s 2022-03-29T17:51:53.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.554473877s (sent 136.563827515s) 2022-03-29T17:51:53.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:53.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:51:53.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 128.562225342s 2022-03-29T17:51:53.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.562225342s (sent 136.564376831s) 2022-03-29T17:51:53.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:53.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:51:53.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 128.558242798s 2022-03-29T17:51:53.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.558242798s (sent 136.564651489s) 2022-03-29T17:51:53.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:53.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,868 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.893 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,869 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:51:53.893 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,869 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.557189941s ub 136.563491821s int 16.000000000s))): start 2022-03-29T17:51:53.893 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,869 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.557189941s ub 136.563491821s int 16.000000000s))): got map 13 2022-03-29T17:51:53.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.557189941s ub 136.563491821s int 16.000000000s))): pg present 2022-03-29T17:51:53.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:51:53.894 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.554473877s ub 136.563827515s int 16.000000000s))): start 2022-03-29T17:51:53.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.557189941s ub 136.563491821s int 16.000000000s) for pg: 2.5 2022-03-29T17:51:53.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 128.557189941s ub 136.563491821s int 16.000000000s) 2022-03-29T17:51:53.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 136.479553223s 2022-03-29T17:51:53.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 136.479553223s 2022-03-29T17:51:53.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.896 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.557189941s ub 136.563491821s int 16.000000000s))): submitting ctx 2022-03-29T17:51:53.896 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,870 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.896 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,871 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.554473877s ub 136.563827515s int 16.000000000s))): got map 13 2022-03-29T17:51:53.896 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,871 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.554473877s ub 136.563827515s int 16.000000000s))): pg present 2022-03-29T17:51:53.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,871 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:51:53.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,871 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.562225342s ub 136.564376831s int 16.000000000s))): start 2022-03-29T17:51:53.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,871 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.554473877s ub 136.563827515s int 16.000000000s) for pg: 2.7 2022-03-29T17:51:53.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 128.554473877s ub 136.563827515s int 16.000000000s) 2022-03-29T17:51:53.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 136.480911255s 2022-03-29T17:51:53.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 136.480911255s 2022-03-29T17:51:53.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.554473877s ub 136.563827515s int 16.000000000s))): submitting ctx 2022-03-29T17:51:53.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.562225342s ub 136.564376831s int 16.000000000s))): got map 13 2022-03-29T17:51:53.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.562225342s ub 136.564376831s int 16.000000000s))): pg present 2022-03-29T17:51:53.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=1 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.557189941s ub 136.563491821s int 16.000000000s))): complete 2022-03-29T17:51:53.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,872 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.562225342s ub 136.564376831s int 16.000000000s) for pg: 2.4 2022-03-29T17:51:53.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 128.562225342s ub 136.564376831s int 16.000000000s) 2022-03-29T17:51:53.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 136.481887817s 2022-03-29T17:51:53.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 136.481887817s 2022-03-29T17:51:53.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.562225342s ub 136.564376831s int 16.000000000s))): submitting ctx 2022-03-29T17:51:53.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.554473877s ub 136.563827515s int 16.000000000s))): complete 2022-03-29T17:51:53.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:51:53.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.558242798s ub 136.564651489s int 16.000000000s))): start 2022-03-29T17:51:53.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,873 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.558242798s ub 136.564651489s int 16.000000000s))): got map 13 2022-03-29T17:51:53.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.558242798s ub 136.564651489s int 16.000000000s))): pg present 2022-03-29T17:51:53.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=1 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.562225342s ub 136.564376831s int 16.000000000s))): complete 2022-03-29T17:51:53.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.558242798s ub 136.564651489s int 16.000000000s) for pg: 2.6 2022-03-29T17:51:53.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 128.558242798s ub 136.564651489s int 16.000000000s) 2022-03-29T17:51:53.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 136.483169556s 2022-03-29T17:51:53.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 136.483169556s 2022-03-29T17:51:53.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.558242798s ub 136.564651489s int 16.000000000s))): submitting ctx 2022-03-29T17:51:53.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,874 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=1 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.1 pg_lease(ru 128.558242798s ub 136.564651489s int 16.000000000s))): complete 2022-03-29T17:51:53.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,875 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:53.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,875 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:53.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,875 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:51:53.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,875 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.492202759s ub 136.498077393s int 16.000000000s))): start 2022-03-29T17:51:53.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,875 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:53.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,876 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:53.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,876 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.492202759s ub 136.498077393s int 16.000000000s))): got map 13 2022-03-29T17:51:53.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,876 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.492202759s ub 136.498077393s int 16.000000000s))): pg present 2022-03-29T17:51:53.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,877 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.492202759s ub 136.498077393s int 16.000000000s) for pg: 2.1 2022-03-29T17:51:53.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,877 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 128.492202759s ub 136.498077393s int 16.000000000s) 2022-03-29T17:51:53.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,877 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 136.573120117s 2022-03-29T17:51:53.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,877 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 136.573120117s 2022-03-29T17:51:53.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,877 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,877 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.492202759s ub 136.498077393s int 16.000000000s))): submitting ctx 2022-03-29T17:51:53.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,877 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,877 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=2 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.2 pg_lease(ru 128.492202759s ub 136.498077393s int 16.000000000s))): complete 2022-03-29T17:51:53.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,892 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:51:53.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,892 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:51:53.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,892 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:51:53.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,893 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 128.497055054s 2022-03-29T17:51:53.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,893 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 128.497055054s (sent 136.501800537s) 2022-03-29T17:51:53.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,893 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,893 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:51:53.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,893 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,894 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:51:53.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,910 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:51:53.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,910 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563491821s))): start 2022-03-29T17:51:53.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,911 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563491821s))): got map 13 2022-03-29T17:51:53.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,911 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563491821s))): pg present 2022-03-29T17:51:53.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563491821s) for pg: 2.5 2022-03-29T17:51:53.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 136.563491821s 2022-03-29T17:51:53.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.563491821s (sent 136.563491821s) 2022-03-29T17:51:53.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563491821s))): submitting ctx 2022-03-29T17:51:53.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:51:53.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563827515s))): start 2022-03-29T17:51:53.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563827515s))): got map 13 2022-03-29T17:51:53.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,912 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563827515s))): pg present 2022-03-29T17:51:53.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563827515s) for pg: 2.7 2022-03-29T17:51:53.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 136.563827515s 2022-03-29T17:51:53.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.563827515s (sent 136.563827515s) 2022-03-29T17:51:53.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563827515s))): submitting ctx 2022-03-29T17:51:53.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:51:53.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564376831s))): start 2022-03-29T17:51:53.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=0 pgid=2.5 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563491821s))): complete 2022-03-29T17:51:53.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564376831s))): got map 13 2022-03-29T17:51:53.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,913 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564376831s))): pg present 2022-03-29T17:51:53.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564376831s) for pg: 2.4 2022-03-29T17:51:53.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 136.564376831s 2022-03-29T17:51:53.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.564376831s (sent 136.564376831s) 2022-03-29T17:51:53.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564376831s))): submitting ctx 2022-03-29T17:51:53.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:51:53.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564651489s))): start 2022-03-29T17:51:53.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,914 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564651489s))): got map 13 2022-03-29T17:51:53.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,915 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564651489s))): pg present 2022-03-29T17:51:53.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,915 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:51:53.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,915 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 128.497055054s ub 136.501800537s int 16.000000000s))): start 2022-03-29T17:51:53.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,915 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564651489s) for pg: 2.6 2022-03-29T17:51:53.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,915 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 136.564651489s 2022-03-29T17:51:53.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,915 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.564651489s (sent 136.564651489s) 2022-03-29T17:51:53.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,915 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,915 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564651489s))): submitting ctx 2022-03-29T17:51:53.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,916 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,916 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=0 pgid=2.7 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.563827515s))): complete 2022-03-29T17:51:53.935 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,916 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:51:53.935 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,916 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.498077393s))): start 2022-03-29T17:51:53.935 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,916 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.498077393s))): got map 13 2022-03-29T17:51:53.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,916 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.498077393s))): pg present 2022-03-29T17:51:53.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,917 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.498077393s) for pg: 2.1 2022-03-29T17:51:53.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,917 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 136.498077393s 2022-03-29T17:51:53.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,917 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.498077393s (sent 136.498077393s) 2022-03-29T17:51:53.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,917 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,917 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.498077393s))): submitting ctx 2022-03-29T17:51:53.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,917 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,917 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 128.497055054s ub 136.501800537s int 16.000000000s))): got map 13 2022-03-29T17:51:53.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,918 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 128.497055054s ub 136.501800537s int 16.000000000s))): pg present 2022-03-29T17:51:53.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,918 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=0 pgid=2.4 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564376831s))): complete 2022-03-29T17:51:53.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:53,918 [shard 0] osd - peering_event(id=91, detail=PeeringEvent(from=1 pgid=2.1 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.498077393s))): complete 2022-03-29T17:51:53.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,918 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 128.497055054s ub 136.501800537s int 16.000000000s) for pg: 2.2 2022-03-29T17:51:53.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,918 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 128.497055054s ub 136.501800537s int 16.000000000s) 2022-03-29T17:51:53.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,918 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 136.614807129s 2022-03-29T17:51:53.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,918 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 136.614807129s 2022-03-29T17:51:53.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,919 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,919 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 128.497055054s ub 136.501800537s int 16.000000000s))): submitting ctx 2022-03-29T17:51:53.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,919 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,920 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=0 pgid=2.6 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.0 pg_lease_ack(ruub 136.564651489s))): complete 2022-03-29T17:51:53.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:53,923 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=0 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLease epoch 13 from osd.0 pg_lease(ru 128.497055054s ub 136.501800537s int 16.000000000s))): complete 2022-03-29T17:51:53.954 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:51:53.954 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.501800537s))): start 2022-03-29T17:51:53.954 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.501800537s))): got map 13 2022-03-29T17:51:53.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.501800537s))): pg present 2022-03-29T17:51:53.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - do_peering_event handling epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.501800537s) for pg: 2.2 2022-03-29T17:51:53.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 136.501800537s 2022-03-29T17:51:53.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.501800537s (sent 136.501800537s) 2022-03-29T17:51:53.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - log is not dirty 2022-03-29T17:51:53.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.501800537s))): submitting ctx 2022-03-29T17:51:53.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,952 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:53.957 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:53,953 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=2.2 sent=13 requested=13 evt=epoch_sent: 13 epoch_requested: 13 MLeaseAck epoch 13 from osd.1 pg_lease_ack(ruub 136.501800537s))): complete 2022-03-29T17:51:54.775 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:54,774 [shard 0] alienstore - stat 2022-03-29T17:51:54.798 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:54,797 [shard 0] alienstore - stat 2022-03-29T17:51:54.799 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:54,797 [shard 0] alienstore - stat 2022-03-29T17:51:55.780 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:55,779 [shard 0] alienstore - stat 2022-03-29T17:51:55.805 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:55,803 [shard 0] alienstore - stat 2022-03-29T17:51:55.806 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:55,803 [shard 0] alienstore - stat 2022-03-29T17:51:56.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:3 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): start 2022-03-29T17:51:56.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:3 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): in repeat 2022-03-29T17:51:56.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:3 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]) same_interval_since: 11 2022-03-29T17:51:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - OpSequencer::start_op: op=2, last_started=1, last_unblocked=1, last_completed=1 2022-03-29T17:51:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - do_recover_missing check for recovery, 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:51:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): got obc lock 2022-03-29T17:51:56.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - with_head_obc 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:51:56.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - with_head_obc: found 2:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:51:56.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:51:56.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,742 [shard 0] osd - handling op omap-get-vals on object 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:51:56.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,743 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:51:56.748 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,743 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8 - object 2:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:51:56.748 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,743 [shard 0] osd - with_head_obc: released 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:51:56.748 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,743 [shard 0] osd - OpSequencer::finish_op_in_order: op=2, last_started=2, last_unblocked=2, last_completed=1 2022-03-29T17:51:56.748 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:56,743 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): destroying 2022-03-29T17:51:56.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:4 2.2 2.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): start 2022-03-29T17:51:56.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:4 2.2 2.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): in repeat 2022-03-29T17:51:56.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:4 2.2 2.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e13) v8]) same_interval_since: 11 2022-03-29T17:51:56.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - OpSequencer::start_op: op=2, last_started=1, last_unblocked=1, last_completed=1 2022-03-29T17:51:56.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - do_recover_missing check for recovery, 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:51:56.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:4 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): got obc lock 2022-03-29T17:51:56.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - with_head_obc 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:51:56.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - with_head_obc: found 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head in cache 2022-03-29T17:51:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - do_osd_ops_execute: object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head - handling op omap-get-vals 2022-03-29T17:51:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - handling op omap-get-vals on object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:51:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - do_osd_ops_execute: object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head all operations successful 2022-03-29T17:51:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:4 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8 - object 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head sending reply 2022-03-29T17:51:56.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,746 [shard 0] osd - with_head_obc: released 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head 2022-03-29T17:51:56.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,747 [shard 0] osd - OpSequencer::finish_op_in_order: op=2, last_started=2, last_unblocked=2, last_completed=1 2022-03-29T17:51:56.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:56,747 [shard 0] osd - client_request(id=2, detail=m=[osd_op(client.4127.0:4 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e13) v8]): destroying 2022-03-29T17:51:56.785 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:56,783 [shard 0] alienstore - stat 2022-03-29T17:51:56.808 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:56,806 [shard 0] alienstore - stat 2022-03-29T17:51:56.809 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:56,807 [shard 0] alienstore - stat 2022-03-29T17:51:57.048 INFO:tasks.workunit.client.0.smithi138.stderr:Note: switching to '60fdd357d1aeaa5f5a03a623dc666183c6bee5d5'. 2022-03-29T17:51:57.049 INFO:tasks.workunit.client.0.smithi138.stderr: 2022-03-29T17:51:57.049 INFO:tasks.workunit.client.0.smithi138.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-29T17:51:57.049 INFO:tasks.workunit.client.0.smithi138.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-29T17:51:57.049 INFO:tasks.workunit.client.0.smithi138.stderr:state without impacting any branches by switching back to a branch. 2022-03-29T17:51:57.050 INFO:tasks.workunit.client.0.smithi138.stderr: 2022-03-29T17:51:57.050 INFO:tasks.workunit.client.0.smithi138.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-29T17:51:57.050 INFO:tasks.workunit.client.0.smithi138.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-29T17:51:57.050 INFO:tasks.workunit.client.0.smithi138.stderr: 2022-03-29T17:51:57.051 INFO:tasks.workunit.client.0.smithi138.stderr: git switch -c 2022-03-29T17:51:57.051 INFO:tasks.workunit.client.0.smithi138.stderr: 2022-03-29T17:51:57.051 INFO:tasks.workunit.client.0.smithi138.stderr:Or undo this operation with: 2022-03-29T17:51:57.051 INFO:tasks.workunit.client.0.smithi138.stderr: 2022-03-29T17:51:57.051 INFO:tasks.workunit.client.0.smithi138.stderr: git switch - 2022-03-29T17:51:57.052 INFO:tasks.workunit.client.0.smithi138.stderr: 2022-03-29T17:51:57.052 INFO:tasks.workunit.client.0.smithi138.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-29T17:51:57.052 INFO:tasks.workunit.client.0.smithi138.stderr: 2022-03-29T17:51:57.052 INFO:tasks.workunit.client.0.smithi138.stderr:HEAD is now at 60fdd357d1a Merge pull request #45471 from amathuria/wip-54601-quincy 2022-03-29T17:51:57.054 DEBUG:teuthology.orchestra.run.smithi138:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-03-29T17:51:57.119 INFO:tasks.workunit.client.0.smithi138.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-03-29T17:51:57.123 INFO:tasks.workunit.client.0.smithi138.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-03-29T17:51:57.127 INFO:tasks.workunit.client.0.smithi138.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-03-29T17:51:57.675 INFO:tasks.workunit.client.0.smithi138.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-03-29T17:51:57.735 INFO:tasks.workunit.client.0.smithi138.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-03-29T17:51:57.772 INFO:tasks.workunit.client.0.smithi138.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-03-29T17:51:57.774 INFO:tasks.workunit.client.0.smithi138.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-03-29T17:51:57.775 INFO:tasks.workunit.client.0.smithi138.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-03-29T17:51:57.789 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:57,788 [shard 0] alienstore - stat 2022-03-29T17:51:57.809 INFO:tasks.workunit.client.0.smithi138.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-03-29T17:51:57.813 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-03-29T17:51:57.814 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-03-29T17:51:57.814 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:57,811 [shard 0] alienstore - stat 2022-03-29T17:51:57.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:57,811 [shard 0] alienstore - stat 2022-03-29T17:51:57.871 INFO:tasks.workunit:Running workunits matching rbd/test_librbd_python.sh on client.0... 2022-03-29T17:51:57.873 INFO:tasks.workunit:Running workunit rbd/test_librbd_python.sh... 2022-03-29T17:51:57.874 DEBUG:teuthology.orchestra.run.smithi138:workunit test rbd/test_librbd_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=60fdd357d1aeaa5f5a03a623dc666183c6bee5d5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2022-03-29T17:51:57.938 INFO:tasks.workunit.client.0.smithi138.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2022-03-29T17:51:58.176 INFO:tasks.workunit.client.0.smithi138.stderr:+ relpath=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind 2022-03-29T17:51:58.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ '[' -n '' ']' 2022-03-29T17:51:58.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ python3 -m nose -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind/test_rbd.py 2022-03-29T17:51:58.277 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,276 [shard 0] osd - handle_osd_map osd_map(13..14 src has 1..14) v4 2022-03-29T17:51:58.278 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,276 [shard 0] osd - handle_osd_map epochs [13..14], i have 13, src has [1..14] 2022-03-29T17:51:58.279 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,276 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e14 {3.5=14,2022-03-29T17:51:58.272165+0000,3.7=14,2022-03-29T17:51:58.272165+0000,3.8=14,2022-03-29T17:51:58.272165+0000,3.e=14,2022-03-29T17:51:58.272165+0000,3.11=14,2022-03-29T17:51:58.272165+0000,3.16=14,2022-03-29T17:51:58.272165+0000,3.18=14,2022-03-29T17:51:58.272165+0000,3.1d=14,2022-03-29T17:51:58.272165+0000,3.1e=14,2022-03-29T17:51:58.272165+0000}) v2): starting 2022-03-29T17:51:58.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.5 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=? pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.281 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,277 [shard 0] osd - compound_peering_request(id=4, detail=pg_create2(e14 {3.0=14,2022-03-29T17:51:58.272165+0000,3.2=14,2022-03-29T17:51:58.272165+0000,3.4=14,2022-03-29T17:51:58.272165+0000,3.b=14,2022-03-29T17:51:58.272165+0000,3.d=14,2022-03-29T17:51:58.272165+0000,3.10=14,2022-03-29T17:51:58.272165+0000,3.13=14,2022-03-29T17:51:58.272165+0000,3.14=14,2022-03-29T17:51:58.272165+0000,3.19=14,2022-03-29T17:51:58.272165+0000,3.1a=14,2022-03-29T17:51:58.272165+0000,3.1c=14,2022-03-29T17:51:58.272165+0000}) v2): starting 2022-03-29T17:51:58.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.0 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=? pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.2 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=? pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.4 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=? pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.7 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=? pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.8 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=? pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.e e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=? pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.11 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=? pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.16 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=? pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.18 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=? pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.1d e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=? pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.1e e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=? pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,277 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e14 {3.5=14,2022-03-29T17:51:58.272165+0000,3.7=14,2022-03-29T17:51:58.272165+0000,3.8=14,2022-03-29T17:51:58.272165+0000,3.e=14,2022-03-29T17:51:58.272165+0000,3.11=14,2022-03-29T17:51:58.272165+0000,3.16=14,2022-03-29T17:51:58.272165+0000,3.18=14,2022-03-29T17:51:58.272165+0000,3.1d=14,2022-03-29T17:51:58.272165+0000,3.1e=14,2022-03-29T17:51:58.272165+0000}) v2): about to fork future 2022-03-29T17:51:58.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,277 [shard 0] osd - handle_pg_create: 3.b e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=? pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - handle_pg_create: 3.d e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=? pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - handle_pg_create: 3.10 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=? pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - handle_pg_create: 3.13 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=? pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - handle_pg_create: 3.14 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=? pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - handle_pg_create: 3.19 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=? pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - handle_pg_create: 3.1a e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=? pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - handle_pg_create: 3.1c e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,278 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=? pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.297 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,278 [shard 0] osd - compound_peering_request(id=4, detail=pg_create2(e14 {3.0=14,2022-03-29T17:51:58.272165+0000,3.2=14,2022-03-29T17:51:58.272165+0000,3.4=14,2022-03-29T17:51:58.272165+0000,3.b=14,2022-03-29T17:51:58.272165+0000,3.d=14,2022-03-29T17:51:58.272165+0000,3.10=14,2022-03-29T17:51:58.272165+0000,3.13=14,2022-03-29T17:51:58.272165+0000,3.14=14,2022-03-29T17:51:58.272165+0000,3.19=14,2022-03-29T17:51:58.272165+0000,3.1a=14,2022-03-29T17:51:58.272165+0000,3.1c=14,2022-03-29T17:51:58.272165+0000}) v2): about to fork future 2022-03-29T17:51:58.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,279 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:51:58.298 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,279 [shard 0] osd - evt epoch is 14, i have 13, will wait 2022-03-29T17:51:58.299 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,279 [shard 0] osd - osdmap_subscribe(13) 2022-03-29T17:51:58.299 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,279 [shard 0] osd - evt epoch is 14, i have 13, will wait 2022-03-29T17:51:58.299 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,279 [shard 0] osd - osdmap_subscribe(13) 2022-03-29T17:51:58.300 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,280 [shard 0] osd - handle_osd_map osd_map(14..14 src has 1..14) v4 2022-03-29T17:51:58.300 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,280 [shard 0] osd - handle_osd_map epochs [14..14], i have 13, src has [1..14] 2022-03-29T17:51:58.302 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,280 [shard 0] osd - compound_peering_request(id=3, detail=pg_create2(e14 {3.1=14,2022-03-29T17:51:58.272165+0000,3.3=14,2022-03-29T17:51:58.272165+0000,3.6=14,2022-03-29T17:51:58.272165+0000,3.9=14,2022-03-29T17:51:58.272165+0000,3.a=14,2022-03-29T17:51:58.272165+0000,3.c=14,2022-03-29T17:51:58.272165+0000,3.f=14,2022-03-29T17:51:58.272165+0000,3.12=14,2022-03-29T17:51:58.272165+0000,3.15=14,2022-03-29T17:51:58.272165+0000,3.17=14,2022-03-29T17:51:58.272165+0000,3.1b=14,2022-03-29T17:51:58.272165+0000,3.1f=14,2022-03-29T17:51:58.272165+0000}) v2): starting 2022-03-29T17:51:58.302 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,280 [shard 0] osd - handle_pg_create: 3.1 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.303 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,280 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=? pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.303 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,280 [shard 0] osd - handle_pg_create: 3.3 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.303 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,280 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=? pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.303 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,280 [shard 0] osd - handle_pg_create: 3.6 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.304 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,281 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=? pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.304 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,281 [shard 0] osd - handle_pg_create: 3.9 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.304 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,281 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=? pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.304 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,281 [shard 0] osd - handle_pg_create: 3.a e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.305 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,281 [shard 0] osd - handle_osd_map osd_map(13..14 src has 1..14) v4 2022-03-29T17:51:58.305 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,281 [shard 0] osd - handle_osd_map epochs [13..14], i have 13, src has [1..14] 2022-03-29T17:51:58.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:51:58.308 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,281 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=? pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.308 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - handle_pg_create: 3.c e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.308 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=? pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.308 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - handle_pg_create: 3.f e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.309 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=? pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.309 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - handle_pg_create: 3.12 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.309 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=? pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.309 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - handle_pg_create: 3.15 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.310 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=? pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.310 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - handle_pg_create: 3.17 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.310 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=? pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.310 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - handle_pg_create: 3.1b e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.310 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=? pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.311 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - handle_pg_create: 3.1f e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.311 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=? pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): start 2022-03-29T17:51:58.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,282 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:51:58.312 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,282 [shard 0] osd - compound_peering_request(id=3, detail=pg_create2(e14 {3.1=14,2022-03-29T17:51:58.272165+0000,3.3=14,2022-03-29T17:51:58.272165+0000,3.6=14,2022-03-29T17:51:58.272165+0000,3.9=14,2022-03-29T17:51:58.272165+0000,3.a=14,2022-03-29T17:51:58.272165+0000,3.c=14,2022-03-29T17:51:58.272165+0000,3.f=14,2022-03-29T17:51:58.272165+0000,3.12=14,2022-03-29T17:51:58.272165+0000,3.15=14,2022-03-29T17:51:58.272165+0000,3.17=14,2022-03-29T17:51:58.272165+0000,3.1b=14,2022-03-29T17:51:58.272165+0000,3.1f=14,2022-03-29T17:51:58.272165+0000}) v2): about to fork future 2022-03-29T17:51:58.313 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,283 [shard 0] osd - osd.2: committed_osd_maps(14, 14) 2022-03-29T17:51:58.313 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,283 [shard 0] osd - osd.2: map e 14 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:51:58.314 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,283 [shard 0] osd - osd.0: committed_osd_maps(14, 14) 2022-03-29T17:51:58.314 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,283 [shard 0] osd - evt epoch is 14, i have 13, will wait 2022-03-29T17:51:58.314 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,283 [shard 0] osd - osdmap_subscribe(13) 2022-03-29T17:51:58.315 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,283 [shard 0] osd - osd.0: map e 14 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:51:58.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_advance_map(id=9, detail=PGAdvanceMap(pg=2.0 from=13 to=14)): start 2022-03-29T17:51:58.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_advance_map(id=10, detail=PGAdvanceMap(pg=2.1 from=13 to=14)): start 2022-03-29T17:51:58.316 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,284 [shard 0] osd - osd.1: committed_osd_maps(14, 14) 2022-03-29T17:51:58.317 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,284 [shard 0] osd - osd.1: map e 14 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:51:58.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_advance_map(id=28, detail=PGAdvanceMap(pg=1.0 from=13 to=14)): start 2022-03-29T17:51:58.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_advance_map(id=29, detail=PGAdvanceMap(pg=2.0 from=13 to=14)): start 2022-03-29T17:51:58.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_advance_map(id=30, detail=PGAdvanceMap(pg=2.1 from=13 to=14)): start 2022-03-29T17:51:58.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_advance_map(id=11, detail=PGAdvanceMap(pg=2.3 from=13 to=14)): start 2022-03-29T17:51:58.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:51:58.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:58.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:58.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:58.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:58.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:58.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:58.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:58.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:51:58.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,284 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=31, detail=PGAdvanceMap(pg=2.2 from=13 to=14)): start 2022-03-29T17:51:58.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=32, detail=PGAdvanceMap(pg=2.3 from=13 to=14)): start 2022-03-29T17:51:58.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=33, detail=PGAdvanceMap(pg=2.4 from=13 to=14)): start 2022-03-29T17:51:58.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=34, detail=PGAdvanceMap(pg=2.5 from=13 to=14)): start 2022-03-29T17:51:58.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=35, detail=PGAdvanceMap(pg=2.6 from=13 to=14)): start 2022-03-29T17:51:58.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=36, detail=PGAdvanceMap(pg=2.7 from=13 to=14)): start 2022-03-29T17:51:58.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:51:58.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:58.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 13 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:51:58.331 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=16, detail=PGAdvanceMap(pg=1.0 from=13 to=14)): start 2022-03-29T17:51:58.331 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=17, detail=PGAdvanceMap(pg=2.2 from=13 to=14)): start 2022-03-29T17:51:58.332 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=18, detail=PGAdvanceMap(pg=2.4 from=13 to=14)): start 2022-03-29T17:51:58.332 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=19, detail=PGAdvanceMap(pg=2.5 from=13 to=14)): start 2022-03-29T17:51:58.332 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=20, detail=PGAdvanceMap(pg=2.6 from=13 to=14)): start 2022-03-29T17:51:58.332 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_advance_map(id=21, detail=PGAdvanceMap(pg=2.7 from=13 to=14)): start 2022-03-29T17:51:58.333 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.333 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:58.333 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.333 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.334 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:58.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:58.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:58.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 13:10 2022-03-29T17:51:58.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:58.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:58.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 13:10 2022-03-29T17:51:58.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:58.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:58.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:51:58.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active advmap 2022-03-29T17:51:58.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Started advmap 2022-03-29T17:51:58.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 13 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:58.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:58.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map 2022-03-29T17:51:58.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,286 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.352 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,285 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.352 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:51:58.353 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.353 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.353 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.353 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.354 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:58.354 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.354 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:51:58.354 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 13 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active state: Started advmap 2022-03-29T17:51:58.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:58.357 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_advance_map(id=9, detail=PGAdvanceMap(pg=2.0 from=13 to=14)): complete 2022-03-29T17:51:58.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_advance_map(id=10, detail=PGAdvanceMap(pg=2.1 from=13 to=14)): complete 2022-03-29T17:51:58.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_advance_map(id=11, detail=PGAdvanceMap(pg=2.3 from=13 to=14)): complete 2022-03-29T17:51:58.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=? pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.360 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,287 [shard 0] osd - osd.2: now active 2022-03-29T17:51:58.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=? pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - Creating 3.5 2022-03-29T17:51:58.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=? pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,287 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: handle ActMap primary 2022-03-29T17:51:58.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean publish_stats_to_osd 13:13 2022-03-29T17:51:58.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 13 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:58.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:58.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:58.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:58.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 13:10 2022-03-29T17:51:58.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:51:58.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.369 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:51:58.369 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map 2022-03-29T17:51:58.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 13 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:51:58.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:58.372 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:58.372 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=? pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.373 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.374 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,289 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:51:58.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:51:58.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.922477722s + 0.000000000s) 2022-03-29T17:51:58.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 13 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:58.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:58.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:58.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:58.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 13:10 2022-03-29T17:51:58.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.381 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.381 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:58.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:58.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,289 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 13:10 2022-03-29T17:51:58.384 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,288 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:51:58.384 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.385 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.385 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.385 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - handle_osd_map osd_map(14..14 src has 1..14) v4 2022-03-29T17:51:58.385 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - handle_osd_map epochs [14..14], i have 14, src has [1..14] 2022-03-29T17:51:58.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:58.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:58.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.387 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 13:10 2022-03-29T17:51:58.387 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.387 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_advance_map(id=12, detail=PGAdvanceMap(pg=3.5 from=14 to=14 do_init)): start 2022-03-29T17:51:58.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=? pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001515 0 0.000000 2022-03-29T17:51:58.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - Exiting state: Initial, entered at 1648576318.2891881, 0.0 spent on 0 events 2022-03-29T17:51:58.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000118 1 2022-03-29T17:51:58.290849+0000 2022-03-29T17:51:58.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - Exiting state: Reset, entered at 1648576318.2907548, 1648576318.2908494 spent on 1 events 2022-03-29T17:51:58.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,290 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 13 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:58.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:58.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:58.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:51:58.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 13 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:58.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:58.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.400 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,290 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.400 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_advance_map(id=20, detail=PGAdvanceMap(pg=2.6 from=13 to=14)): complete 2022-03-29T17:51:58.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_advance_map(id=21, detail=PGAdvanceMap(pg=2.7 from=13 to=14)): complete 2022-03-29T17:51:58.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_advance_map(id=19, detail=PGAdvanceMap(pg=2.5 from=13 to=14)): complete 2022-03-29T17:51:58.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_advance_map(id=18, detail=PGAdvanceMap(pg=2.4 from=13 to=14)): complete 2022-03-29T17:51:58.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_advance_map(id=16, detail=PGAdvanceMap(pg=1.0 from=13 to=14)): complete 2022-03-29T17:51:58.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,291 [shard 0] osd - pg_advance_map(id=17, detail=PGAdvanceMap(pg=2.2 from=13 to=14)): complete 2022-03-29T17:51:58.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.001068 0 0.000000 2022-03-29T17:51:58.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,292 [shard 0] osd - Exiting state: Start, entered at 1648576318.2909365, 0.0 spent on 0 events 2022-03-29T17:51:58.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,292 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,292 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.406 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.406 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,292 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - Created 3.5 2022-03-29T17:51:58.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:58.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:58.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 13:10 2022-03-29T17:51:58.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 13 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:58.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:58.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:58.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,292 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:58.416 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,293 [shard 0] osd - osd.0: now active 2022-03-29T17:51:58.416 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=? pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.416 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - Creating 3.17 2022-03-29T17:51:58.416 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=? pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.417 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=? pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.417 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=? pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.417 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=? pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,293 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.419 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,293 [shard 0] osd - send_alive want=14 up_thru_wanted=11 2022-03-29T17:51:58.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - send_alive up_thru_wanted=14 up_thru=11 2022-03-29T17:51:58.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:58.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:58.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:58.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 13:10 2022-03-29T17:51:58.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 14 2022-03-29T17:51:58.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - pg_advance_map(id=28, detail=PGAdvanceMap(pg=1.0 from=13 to=14)): complete 2022-03-29T17:51:58.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,293 [shard 0] osd - pg_advance_map(id=31, detail=PGAdvanceMap(pg=2.2 from=13 to=14)): complete 2022-03-29T17:51:58.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=? pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.424 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,294 [shard 0] osd - pg_epoch 14 pg[3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.425 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,294 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - pg_epoch 14 pg[3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:58.426 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.426 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:58.426 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.903266907s + 0.000000000s) 2022-03-29T17:51:58.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=? pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=? pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - Creating 3.e 2022-03-29T17:51:58.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,294 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=? pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.428 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,294 [shard 0] osd - pg_epoch 14 pg[3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.428 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,294 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_advance_map(id=30, detail=PGAdvanceMap(pg=2.1 from=13 to=14)): complete 2022-03-29T17:51:58.431 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.431 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_advance_map(id=22, detail=PGAdvanceMap(pg=3.17 from=14 to=14 do_init)): start 2022-03-29T17:51:58.431 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=? pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.431 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=? pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.432 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001114 0 0.000000 2022-03-29T17:51:58.432 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,295 [shard 0] osd - Exiting state: Initial, entered at 1648576318.2941647, 0.0 spent on 0 events 2022-03-29T17:51:58.432 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.432 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,295 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.433 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.433 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.433 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.433 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.433 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.434 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000128 1 2022-03-29T17:51:58.295432+0000 2022-03-29T17:51:58.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_advance_map(id=32, detail=PGAdvanceMap(pg=2.3 from=13 to=14)): complete 2022-03-29T17:51:58.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_advance_map(id=34, detail=PGAdvanceMap(pg=2.5 from=13 to=14)): complete 2022-03-29T17:51:58.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=? pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 0} acting {2, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 0}+{2, 0} 2022-03-29T17:51:58.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000123930} 2022-03-29T17:51:58.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,295 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.928932190s + 0.000000000s) 2022-03-29T17:51:58.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_advance_map(id=33, detail=PGAdvanceMap(pg=2.4 from=13 to=14)): complete 2022-03-29T17:51:58.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_advance_map(id=29, detail=PGAdvanceMap(pg=2.0 from=13 to=14)): complete 2022-03-29T17:51:58.441 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_advance_map(id=35, detail=PGAdvanceMap(pg=2.6 from=13 to=14)): complete 2022-03-29T17:51:58.441 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_advance_map(id=36, detail=PGAdvanceMap(pg=2.7 from=13 to=14)): complete 2022-03-29T17:51:58.441 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - osd.1: now active 2022-03-29T17:51:58.441 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=? pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.441 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - Creating 3.1a 2022-03-29T17:51:58.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=? pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=? pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.444 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,295 [shard 0] osd - Exiting state: Reset, entered at 1648576318.2953267, 1648576318.2954323 spent on 1 events 2022-03-29T17:51:58.445 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.445 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.445 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.445 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.446 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.446 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000069 0 0.000000 2022-03-29T17:51:58.446 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - Exiting state: Start, entered at 1648576318.2965574, 0.0 spent on 0 events 2022-03-29T17:51:58.446 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.446 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.447 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.447 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.447 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.447 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,296 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,296 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_advance_map(id=13, detail=PGAdvanceMap(pg=3.e from=14 to=14 do_init)): start 2022-03-29T17:51:58.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.5 2022-03-29T17:51:58.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=? pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002462 0 0.000000 2022-03-29T17:51:58.453 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,297 [shard 0] osd - Exiting state: Initial, entered at 1648576318.2949336, 0.0 spent on 0 events 2022-03-29T17:51:58.453 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.453 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,297 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.455 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.455 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,297 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.455 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.456 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.456 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:51:58.456 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.456 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:51:58.456 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.993667603s + 0.000000000s) 2022-03-29T17:51:58.457 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.457 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_advance_map(id=37, detail=PGAdvanceMap(pg=3.1a from=14 to=14 do_init)): start 2022-03-29T17:51:58.457 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=? pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.457 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001471 0 0.000000 2022-03-29T17:51:58.458 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,297 [shard 0] osd - Exiting state: Initial, entered at 1648576318.2964706, 0.0 spent on 0 events 2022-03-29T17:51:58.459 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - Created 3.17 2022-03-29T17:51:58.461 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.461 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - send_alive want=14 up_thru_wanted=11 2022-03-29T17:51:58.462 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - send_alive up_thru_wanted=14 up_thru=11 2022-03-29T17:51:58.462 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=? pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.462 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=? pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,297 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.466 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001170 1 2022-03-29T17:51:58.298574+0000 2022-03-29T17:51:58.466 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - Exiting state: Reset, entered at 1648576318.2974436, 1648576318.2985744 spent on 1 events 2022-03-29T17:51:58.467 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.467 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.467 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.467 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.468 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.468 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000042 0 0.000000 2022-03-29T17:51:58.468 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - Exiting state: Start, entered at 1648576318.2986817, 0.0 spent on 0 events 2022-03-29T17:51:58.468 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.468 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.469 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.469 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.469 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.469 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,298 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,2 down blocked_by {} 2022-03-29T17:51:58.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:51:58.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,298 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.474 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.474 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.474 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.475 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000180 1 2022-03-29T17:51:58.299154+0000 2022-03-29T17:51:58.476 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - Exiting state: Reset, entered at 1648576318.2989967, 1648576318.299155 spent on 1 events 2022-03-29T17:51:58.476 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.476 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.476 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.476 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000047 0 0.000000 2022-03-29T17:51:58.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - Exiting state: Start, entered at 1648576318.2992392, 0.0 spent on 0 events 2022-03-29T17:51:58.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,299 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:58.480 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - Creating 3.9 2022-03-29T17:51:58.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.481 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,299 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=? pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - Created 3.e 2022-03-29T17:51:58.485 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,300 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.486 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,300 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=? pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - Creating 3.16 2022-03-29T17:51:58.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_advance_map(id=12, detail=PGAdvanceMap(pg=3.5 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.490 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.490 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.490 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.490 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.490 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.491 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.491 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - Created 3.1a 2022-03-29T17:51:58.491 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,300 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.491 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.492 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,300 [shard 0] osd - send_alive want=14 up_thru_wanted=11 2022-03-29T17:51:58.492 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - send_alive up_thru_wanted=14 up_thru=11 2022-03-29T17:51:58.492 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=? pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.492 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,300 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=? pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.493 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,301 [shard 0] osd - pg_epoch 14 pg[3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.494 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,301 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - pg_epoch 14 pg[3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 2} acting {0, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 2}+{0, 2} 2022-03-29T17:51:58.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000145510} 2022-03-29T17:51:58.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.910293579s + 0.000000000s) 2022-03-29T17:51:58.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.5 2022-03-29T17:51:58.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.498 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,301 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,301 [shard 0] osd - peering_event(id=92, detail=PeeringEvent(from=? pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=? pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - Creating 3.14 2022-03-29T17:51:58.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.1a 2022-03-29T17:51:58.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=? pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=? pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - pg_advance_map(id=23, detail=PGAdvanceMap(pg=3.9 from=14 to=14 do_init)): start 2022-03-29T17:51:58.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.17 2022-03-29T17:51:58.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=? pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,302 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.503 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,302 [shard 0] osd - pg_epoch 14 pg[3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.504 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,302 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.504 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001861 0 0.000000 2022-03-29T17:51:58.505 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3011606, 0.0 spent on 0 events 2022-03-29T17:51:58.505 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=? pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.506 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.506 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.506 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.506 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.509 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.509 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.511 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000625 1 2022-03-29T17:51:58.303680+0000 2022-03-29T17:51:58.511 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3030794, 1648576318.3036802 spent on 1 events 2022-03-29T17:51:58.511 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.512 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.512 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.512 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.512 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.512 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000040 0 0.000000 2022-03-29T17:51:58.513 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Exiting state: Start, entered at 1648576318.3037822, 0.0 spent on 0 events 2022-03-29T17:51:58.513 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.513 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.513 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.513 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.514 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.514 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,303 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,2 down blocked_by {} 2022-03-29T17:51:58.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,303 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:51:58.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:51:58.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.937210083s + 0.000000000s) 2022-03-29T17:51:58.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_advance_map(id=14, detail=PGAdvanceMap(pg=3.16 from=14 to=14 do_init)): start 2022-03-29T17:51:58.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=? pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:51:58.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:51:58.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.000686646s + 0.000000000s) 2022-03-29T17:51:58.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - pg_advance_map(id=38, detail=PGAdvanceMap(pg=3.14 from=14 to=14 do_init)): start 2022-03-29T17:51:58.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,304 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:51:58.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:58.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_advance_map(id=22, detail=PGAdvanceMap(pg=3.17 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - Created 3.9 2022-03-29T17:51:58.527 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.527 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.e 2022-03-29T17:51:58.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=? pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.531 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002784 0 0.000000 2022-03-29T17:51:58.531 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - Exiting state: Initial, entered at 1648576318.302877, 0.0 spent on 0 events 2022-03-29T17:51:58.531 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.532 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.533 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000113 1 2022-03-29T17:51:58.305803+0000 2022-03-29T17:51:58.533 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - Exiting state: Reset, entered at 1648576318.305712, 1648576318.3058035 spent on 1 events 2022-03-29T17:51:58.533 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.534 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.534 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.534 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.534 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000049 0 0.000000 2022-03-29T17:51:58.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - Exiting state: Start, entered at 1648576318.3058858, 0.0 spent on 0 events 2022-03-29T17:51:58.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,305 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.5 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:58.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=? pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.17 2022-03-29T17:51:58.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=? pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.541 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003146 0 0.000000 2022-03-29T17:51:58.541 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Exiting state: Initial, entered at 1648576318.30343, 0.0 spent on 0 events 2022-03-29T17:51:58.541 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.542 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.543 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000113 1 2022-03-29T17:51:58.306741+0000 2022-03-29T17:51:58.543 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3066475, 1648576318.306741 spent on 1 events 2022-03-29T17:51:58.543 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.544 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.544 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.544 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.544 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000036 0 0.000000 2022-03-29T17:51:58.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Exiting state: Start, entered at 1648576318.3068192, 0.0 spent on 0 events 2022-03-29T17:51:58.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.546 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.546 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.546 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,306 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.549 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.549 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,307 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.549 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.549 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,307 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - Created 3.16 2022-03-29T17:51:58.553 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,307 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.554 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.554 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,306 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:58.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,307 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.558 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,308 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.560 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - Creating 3.6 2022-03-29T17:51:58.560 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=? pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.560 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,308 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.560 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.560 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=? pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.561 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.9 2022-03-29T17:51:58.561 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.561 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=? pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.561 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - pg_advance_map(id=13, detail=PGAdvanceMap(pg=3.e from=14 to=14 do_init)): complete 2022-03-29T17:51:58.563 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,308 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=? pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - Creating 3.1e 2022-03-29T17:51:58.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - peering_event(id=101, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,308 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - Created 3.14 2022-03-29T17:51:58.567 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,309 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.567 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,309 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=? pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=? pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.e 2022-03-29T17:51:58.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:58.571 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,309 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.? v2:172.21.15.138:6804/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6804/35577, peer_name=osd.?, cc=10389299161227899038 policy(lossy=false, server=false, standby=true, resetcheck=false) 2022-03-29T17:51:58.572 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,309 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.? v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:51:58.572 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,309 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=? pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.574 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,310 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.575 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,310 [shard 0] osd - pg_epoch 14 pg[3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.575 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,310 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - pg_advance_map(id=23, detail=PGAdvanceMap(pg=3.9 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.575 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.9 2022-03-29T17:51:58.576 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.576 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.576 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.576 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.577 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:58.577 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,310 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=? pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): got map 14 2022-03-29T17:51:58.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - Creating 3.10 2022-03-29T17:51:58.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.5 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:58.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,310 [shard 0] osd - pg_advance_map(id=37, detail=PGAdvanceMap(pg=3.1a from=14 to=14 do_init)): complete 2022-03-29T17:51:58.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,309 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - peering_event(id=95, detail=PeeringEvent(from=? pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.581 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,310 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.? v2:172.21.15.138:6802/35579] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6802/35579, peer_name=osd.?, cc=15047428695210124814 policy(lossy=false, server=false, standby=true, resetcheck=false) 2022-03-29T17:51:58.581 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,311 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.? v2:172.21.15.138:6802/35579] write_event: delay ... 2022-03-29T17:51:58.582 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - pg_epoch 14 pg[3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.582 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.582 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:58.582 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.583 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:58.583 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.920761108s + 0.000000000s) 2022-03-29T17:51:58.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1a 2022-03-29T17:51:58.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:58.586 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,312 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.5 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:58.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.5 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:58.588 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,311 [shard 0] osd - pg_epoch 14 pg[3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.588 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,312 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - pg_epoch 14 pg[3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:51:58.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:51:58.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.945831299s + 0.000000000s) 2022-03-29T17:51:58.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - pg_advance_map(id=15, detail=PGAdvanceMap(pg=3.1e from=14 to=14 do_init)): start 2022-03-29T17:51:58.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.16 2022-03-29T17:51:58.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,312 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=? pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,311 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - pg_advance_map(id=24, detail=PGAdvanceMap(pg=3.6 from=14 to=14 do_init)): start 2022-03-29T17:51:58.593 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,313 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.593 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,313 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> unknown.? -@61625] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:61625/0 2022-03-29T17:51:58.594 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,313 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003031 0 0.000000 2022-03-29T17:51:58.594 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,313 [shard 0] osd - Exiting state: Initial, entered at 1648576318.310251, 0.0 spent on 0 events 2022-03-29T17:51:58.594 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,313 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.594 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,313 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:58.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=? pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.14 2022-03-29T17:51:58.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=? pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,313 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.5 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:58.598 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,313 [shard 0] osd - pg_epoch 14 pg[3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.598 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,313 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - pg_advance_map(id=14, detail=PGAdvanceMap(pg=3.16 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.600 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.601 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001004 1 2022-03-29T17:51:58.313440+0000 2022-03-29T17:51:58.602 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3133414, 1648576318.3134408 spent on 1 events 2022-03-29T17:51:58.602 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.602 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.602 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.603 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.603 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.603 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000042 0 0.000000 2022-03-29T17:51:58.603 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - Exiting state: Start, entered at 1648576318.3144386, 0.0 spent on 0 events 2022-03-29T17:51:58.604 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.604 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.604 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.604 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:51:58.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:51:58.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.010757446s + 0.000000000s) 2022-03-29T17:51:58.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,314 [shard 0] osd - pg_advance_map(id=39, detail=PGAdvanceMap(pg=3.10 from=14 to=14 do_init)): start 2022-03-29T17:51:58.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> unknown.? -@54393] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:54393/0 2022-03-29T17:51:58.608 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,314 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.609 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,315 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - peering_event(id=102, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.16 2022-03-29T17:51:58.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - Created 3.6 2022-03-29T17:51:58.617 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,315 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.617 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,315 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,315 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.619 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,316 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - pg_advance_map(id=38, detail=PGAdvanceMap(pg=3.14 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - handle_peering_op on 3.17 from 0 2022-03-29T17:51:58.619 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.17 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:58.621 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,316 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - peering_event(id=97, detail=PeeringEvent(from=? pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.622 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,316 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.005327 0 0.000000 2022-03-29T17:51:58.622 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,316 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3114564, 0.0 spent on 0 events 2022-03-29T17:51:58.622 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,316 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.622 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,316 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,316 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=? pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=? pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.627 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003290 0 0.000000 2022-03-29T17:51:58.627 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - Exiting state: Initial, entered at 1648576318.313869, 0.0 spent on 0 events 2022-03-29T17:51:58.627 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.628 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.629 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000128 1 2022-03-29T17:51:58.317318+0000 2022-03-29T17:51:58.629 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3172104, 1648576318.3173182 spent on 1 events 2022-03-29T17:51:58.629 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:51:58.631 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - Exiting state: Start, entered at 1648576318.317398, 0.0 spent on 0 events 2022-03-29T17:51:58.631 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.632 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - Creating 3.1b 2022-03-29T17:51:58.632 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,317 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.632 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.633 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,317 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.637 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000134 1 2022-03-29T17:51:58.316934+0000 2022-03-29T17:51:58.637 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Exiting state: Reset, entered at 1648576318.316835, 1648576318.3169348 spent on 1 events 2022-03-29T17:51:58.637 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.638 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.638 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.638 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.638 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.639 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000063 0 0.000000 2022-03-29T17:51:58.639 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Exiting state: Start, entered at 1648576318.3180652, 0.0 spent on 0 events 2022-03-29T17:51:58.639 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.639 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.639 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.640 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.641 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.641 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.646 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.646 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.646 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.647 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.647 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:51:58.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=? pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,318 [shard 0] osd - Created 3.10 2022-03-29T17:51:58.652 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - Created 3.1e 2022-03-29T17:51:58.654 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,319 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.655 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,319 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.656 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.656 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,319 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:58.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:58.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.928726196s + 0.000000000s) 2022-03-29T17:51:58.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,319 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.659 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,318 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.660 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,320 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.660 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,320 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=? pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.663 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,320 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - peering_event(id=103, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=? pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,320 [shard 0] osd - Creating 3.8 2022-03-29T17:51:58.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - pg_advance_map(id=25, detail=PGAdvanceMap(pg=3.1b from=14 to=14 do_init)): start 2022-03-29T17:51:58.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.666 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001710 0 0.000000 2022-03-29T17:51:58.667 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,321 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3195221, 0.0 spent on 0 events 2022-03-29T17:51:58.667 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.667 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,321 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.669 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000116 1 2022-03-29T17:51:58.321396+0000 2022-03-29T17:51:58.669 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,321 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3213031, 1648576318.3213966 spent on 1 events 2022-03-29T17:51:58.669 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.669 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,321 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.14 2022-03-29T17:51:58.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:58.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.17 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:58.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,321 [shard 0] osd - Creating 3.2 2022-03-29T17:51:58.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:51:58.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1a from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:58.676 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.676 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.676 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.677 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.001081 0 0.000000 2022-03-29T17:51:58.677 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - Exiting state: Start, entered at 1648576318.3214812, 0.0 spent on 0 events 2022-03-29T17:51:58.677 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.677 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.678 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.678 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.678 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.678 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,322 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,322 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.681 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,323 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.17 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:58.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,323 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.17 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:58.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,323 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:58.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,323 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=0 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.17 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:58.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,323 [shard 0] osd - pg_advance_map(id=15, detail=PGAdvanceMap(pg=3.1e from=14 to=14 do_init)): complete 2022-03-29T17:51:58.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,323 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.1e 2022-03-29T17:51:58.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,323 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,323 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=? pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,323 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.684 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,323 [shard 0] osd - pg_epoch 14 pg[3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.685 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,323 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - Created 3.1b 2022-03-29T17:51:58.688 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,324 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.689 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,324 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.6 2022-03-29T17:51:58.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=? pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=? pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.692 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,324 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_advance_map(id=39, detail=PGAdvanceMap(pg=3.10 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.692 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.693 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,324 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:51:58.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,324 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:51:58.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_epoch 14 pg[3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:51:58.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:51:58.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.958259583s + 0.000000000s) 2022-03-29T17:51:58.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_advance_map(id=16, detail=PGAdvanceMap(pg=3.8 from=14 to=14 do_init)): start 2022-03-29T17:51:58.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1a from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:58.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - Creating 3.1f 2022-03-29T17:51:58.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.020858765s + 0.000000000s) 2022-03-29T17:51:58.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - pg_advance_map(id=40, detail=PGAdvanceMap(pg=3.2 from=14 to=14 do_init)): start 2022-03-29T17:51:58.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.10 2022-03-29T17:51:58.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,325 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=? pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.702 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002251 0 0.000000 2022-03-29T17:51:58.702 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3239381, 0.0 spent on 0 events 2022-03-29T17:51:58.703 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.703 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.704 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000120 1 2022-03-29T17:51:58.326354+0000 2022-03-29T17:51:58.705 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3262553, 1648576318.326354 spent on 1 events 2022-03-29T17:51:58.705 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.705 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.706 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.706 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.706 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - pg_advance_map(id=24, detail=PGAdvanceMap(pg=3.6 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.708 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.6 2022-03-29T17:51:58.708 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,326 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002360 0 0.000000 2022-03-29T17:51:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3246832, 0.0 spent on 0 events 2022-03-29T17:51:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000116 1 2022-03-29T17:51:58.327200+0000 2022-03-29T17:51:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3271046, 1648576318.327201 spent on 1 events 2022-03-29T17:51:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.714 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000036 0 0.000000 2022-03-29T17:51:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Exiting state: Start, entered at 1648576318.3272965, 0.0 spent on 0 events 2022-03-29T17:51:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.717 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:51:58.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Exiting state: Start, entered at 1648576318.32644, 0.0 spent on 0 events 2022-03-29T17:51:58.720 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.721 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.721 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.721 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.721 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.722 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,327 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.722 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.722 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.722 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - Created 3.8 2022-03-29T17:51:58.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,328 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=? pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.728 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,328 [shard 0] osd - pg_epoch 14 pg[3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.729 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,328 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,327 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:58.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.10 2022-03-29T17:51:58.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:58.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,328 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.735 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,329 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.735 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,329 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.736 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.736 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,329 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1a from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:58.736 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1a from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:58.736 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.737 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=? pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.737 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - peering_event(id=105, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1a from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:58.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - pg_epoch 14 pg[3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:58.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:58.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.938705444s + 0.000000000s) 2022-03-29T17:51:58.740 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.740 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - pg_advance_map(id=26, detail=PGAdvanceMap(pg=3.1f from=14 to=14 do_init)): start 2022-03-29T17:51:58.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - Created 3.2 2022-03-29T17:51:58.743 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,330 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.743 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,330 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:51:58.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.16 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:58.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=? pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,329 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1e 2022-03-29T17:51:58.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.747 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,330 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.748 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,330 [shard 0] osd - peering_event(id=100, detail=PeeringEvent(from=? pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.1b 2022-03-29T17:51:58.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.750 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=? pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.750 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.750 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,331 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002870 0 0.000000 2022-03-29T17:51:58.750 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,331 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3284357, 0.0 spent on 0 events 2022-03-29T17:51:58.751 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,331 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.751 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,331 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.753 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,331 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,331 [shard 0] osd - Creating 3.d 2022-03-29T17:51:58.754 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] monc - get_auth_request(con=[osd.2(cluster) v2:172.21.15.138:6802/35579@61625 >> osd.? v2:172.21.15.138:6804/35577], auth_method=0) 2022-03-29T17:51:58.755 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - Creating 3.1d 2022-03-29T17:51:58.758 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001136 1 2022-03-29T17:51:58.331462+0000 2022-03-29T17:51:58.758 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - Exiting state: Reset, entered at 1648576318.331367, 1648576318.3314624 spent on 1 events 2022-03-29T17:51:58.758 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.759 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.759 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.759 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.759 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.760 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000044 0 0.000000 2022-03-29T17:51:58.760 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - Exiting state: Start, entered at 1648576318.3326137, 0.0 spent on 0 events 2022-03-29T17:51:58.760 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.760 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.761 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.761 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.761 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.761 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,332 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,332 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.2 2022-03-29T17:51:58.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=? pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=? pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.766 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,333 [shard 0] osd - pg_epoch 14 pg[3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.766 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,333 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.767 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,333 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - handle_peering_op on 3.5 from 1 2022-03-29T17:51:58.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.5 from 1 notify: (query:14 sent:14 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:58.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - peering_event(id=104, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,333 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - Created 3.1f 2022-03-29T17:51:58.774 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,334 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.775 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,334 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=? pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.775 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,334 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.776 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.776 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,334 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.? -@61625] UPDATE: peer_type=osd, policy(lossy=false server=false standby=true resetcheck=false) 2022-03-29T17:51:58.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:51:58.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:51:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.030693054s + 0.000000000s) 2022-03-29T17:51:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,334 [shard 0] osd - pg_advance_map(id=41, detail=PGAdvanceMap(pg=3.d from=14 to=14 do_init)): start 2022-03-29T17:51:58.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - pg_advance_map(id=16, detail=PGAdvanceMap(pg=3.8 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - Creating 3.c 2022-03-29T17:51:58.781 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - pg_advance_map(id=25, detail=PGAdvanceMap(pg=3.1b from=14 to=14 do_init)): complete 2022-03-29T17:51:58.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.782 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.8 2022-03-29T17:51:58.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.784 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.16 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:58.784 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002356 0 0.000000 2022-03-29T17:51:58.785 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3333886, 0.0 spent on 0 events 2022-03-29T17:51:58.785 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.785 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.785 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.786 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000123 1 2022-03-29T17:51:58.335895+0000 2022-03-29T17:51:58.787 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3358, 1648576318.335896 spent on 1 events 2022-03-29T17:51:58.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,335 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=? pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.789 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,336 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.? -@54393] UPDATE: peer_type=osd, policy(lossy=false server=false standby=true resetcheck=false) 2022-03-29T17:51:58.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - pg_epoch 14 pg[3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 0} acting {2, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 0}+{2, 0} 2022-03-29T17:51:58.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000123930} 2022-03-29T17:51:58.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.969490051s + 0.000000000s) 2022-03-29T17:51:58.791 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,336 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=? pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.792 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,336 [shard 0] monc - get_auth_request(con=[osd.0(cluster) v2:172.21.15.138:6804/35577@54393 >> osd.? v2:172.21.15.138:6802/35579], auth_method=0) 2022-03-29T17:51:58.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1b 2022-03-29T17:51:58.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,336 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.795 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,335 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.795 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.795 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.795 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.796 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.796 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000049 0 0.000000 2022-03-29T17:51:58.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_advance_map(id=17, detail=PGAdvanceMap(pg=3.1d from=14 to=14 do_init)): start 2022-03-29T17:51:58.798 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001877 0 0.000000 2022-03-29T17:51:58.798 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3354278, 0.0 spent on 0 events 2022-03-29T17:51:58.798 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.799 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.803 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - Exiting state: Start, entered at 1648576318.3370218, 0.0 spent on 0 events 2022-03-29T17:51:58.803 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.803 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.803 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.804 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.804 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.804 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,337 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.804 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:51:58.805 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.806 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.807 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.808 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,337 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.1f 2022-03-29T17:51:58.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=? pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.813 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000117 1 2022-03-29T17:51:58.337453+0000 2022-03-29T17:51:58.813 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3373556, 1648576318.3374531 spent on 1 events 2022-03-29T17:51:58.813 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.814 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.814 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.814 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.814 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000062 0 0.000000 2022-03-29T17:51:58.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - Exiting state: Start, entered at 1648576318.3385746, 0.0 spent on 0 events 2022-03-29T17:51:58.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.816 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.816 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.816 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,338 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.816 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,2 down blocked_by {} 2022-03-29T17:51:58.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:51:58.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,338 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - Created 3.d 2022-03-29T17:51:58.819 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,339 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.820 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,339 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.821 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,339 [shard 0] osd - pg_epoch 14 pg[3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.821 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,339 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.822 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,339 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - pg_epoch 14 pg[3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - pg_advance_map(id=40, detail=PGAdvanceMap(pg=3.2 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.824 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,339 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.16 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:58.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.16 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:58.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:58.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.16 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:58.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.2 2022-03-29T17:51:58.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.827 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.827 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.828 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.828 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.828 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.828 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.829 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - Created 3.1d 2022-03-29T17:51:58.829 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,340 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.829 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.829 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,340 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.830 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.830 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.5 from 1 notify: (query:14 sent:14 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:58.830 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,340 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.830 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=? pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.8 2022-03-29T17:51:58.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.831 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,339 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:58.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:58.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.949447632s + 0.000000000s) 2022-03-29T17:51:58.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - pg_advance_map(id=27, detail=PGAdvanceMap(pg=3.c from=14 to=14 do_init)): start 2022-03-29T17:51:58.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - handle_peering_op on 3.17 from 1 2022-03-29T17:51:58.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.17 from 1 notify: (query:14 sent:14 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:58.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,340 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:58.837 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.837 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.838 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.838 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - peering_event(id=94, detail=PeeringEvent(from=? pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=? pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.840 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002517 0 0.000000 2022-03-29T17:51:58.840 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3392146, 0.0 spent on 0 events 2022-03-29T17:51:58.841 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.841 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.842 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000120 1 2022-03-29T17:51:58.341889+0000 2022-03-29T17:51:58.843 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3417914, 1648576318.3418891 spent on 1 events 2022-03-29T17:51:58.843 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.843 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,342 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=? pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.844 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,342 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.5 from 1 notify: (query:14 sent:14 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:58.844 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,342 [shard 0] osd - Creating 3.11 2022-03-29T17:51:58.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,342 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,342 [shard 0] osd - Creating 3.0 2022-03-29T17:51:58.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,342 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.846 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,342 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,342 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.849 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,341 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.849 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.849 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.849 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.001079 0 0.000000 2022-03-29T17:51:58.850 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - Exiting state: Start, entered at 1648576318.341973, 0.0 spent on 0 events 2022-03-29T17:51:58.850 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.850 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.850 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.851 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.851 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.851 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,343 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.853 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - handle_peering_op on 3.6 from 0 2022-03-29T17:51:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.6 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.d 2022-03-29T17:51:58.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=? pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,343 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.858 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.858 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,344 [shard 0] osd - pg_epoch 14 pg[3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.858 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,344 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.858 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.1d 2022-03-29T17:51:58.859 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.859 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=? pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.859 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.859 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,344 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.859 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.5 from 1 notify: (query:14 sent:14 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.5 2022-03-29T17:51:58.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:58.861 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,344 [shard 0] monc - added challenge on [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.? -@61625] 2022-03-29T17:51:58.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - pg_advance_map(id=26, detail=PGAdvanceMap(pg=3.1f from=14 to=14 do_init)): complete 2022-03-29T17:51:58.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,344 [shard 0] osd - Created 3.c 2022-03-29T17:51:58.864 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.864 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,345 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:58.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:58.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:58.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:58.867 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.053508 4 2022-03-29T17:51:58.303235+0000 2022-03-29T17:51:58.867 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,345 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.2921264, 1648576318.3032358 spent on 4 events 2022-03-29T17:51:58.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:58.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,345 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:58.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:58.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:58.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:58.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:58.869 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:58.869 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:58.870 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:58.870 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.1 (up) accepted 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:58.870 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:58.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:58.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 1} async_recovery= 2022-03-29T17:51:58.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:58.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:58.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:58.873 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,346 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.873 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,346 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=? pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1f 2022-03-29T17:51:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,345 [shard 0] osd - pg_epoch 14 pg[3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:51:58.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.877 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:51:58.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.042541504s + 0.000000000s) 2022-03-29T17:51:58.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - pg_advance_map(id=42, detail=PGAdvanceMap(pg=3.0 from=14 to=14 do_init)): start 2022-03-29T17:51:58.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,346 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.6 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:58.879 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,346 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001658 0 0.000000 2022-03-29T17:51:58.879 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,346 [shard 0] osd - Exiting state: Initial, entered at 1648576318.345283, 0.0 spent on 0 events 2022-03-29T17:51:58.881 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.001312 0 0.000000 2022-03-29T17:51:58.881 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.3456886, 0.0 spent on 0 events 2022-03-29T17:51:58.881 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:58.882 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:58.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:58.882 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000076 0 0.000000 2022-03-29T17:51:58.882 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.3470623, 0.0 spent on 0 events 2022-03-29T17:51:58.883 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:58.883 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:58.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.5 from 1 notify: (query:14 sent:14 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:58.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:51:58.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:51:58.886 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,347 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - Creating 3.f 2022-03-29T17:51:58.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.17 from 1 notify: (query:14 sent:14 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:58.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,347 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=? pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001166 1 2022-03-29T17:51:58.348126+0000 2022-03-29T17:51:58.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Exiting state: Reset, entered at 1648576318.346984, 1648576318.348127 spent on 1 events 2022-03-29T17:51:58.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.893 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.893 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.893 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.893 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.893 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000051 0 0.000000 2022-03-29T17:51:58.894 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Exiting state: Start, entered at 1648576318.3482165, 0.0 spent on 0 events 2022-03-29T17:51:58.894 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.894 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.894 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.895 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.895 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.895 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:58.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.980522156s + 0.000000000s) 2022-03-29T17:51:58.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_advance_map(id=18, detail=PGAdvanceMap(pg=3.11 from=14 to=14 do_init)): start 2022-03-29T17:51:58.899 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - peering_event(id=107, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.899 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004617 0 0.000000 2022-03-29T17:51:58.900 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3442295, 0.0 spent on 0 events 2022-03-29T17:51:58.901 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.901 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,348 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,348 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.903 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.904 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,349 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.17 from 1 notify: (query:14 sent:14 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:58.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.904 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,349 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_advance_map(id=41, detail=PGAdvanceMap(pg=3.d from=14 to=14 do_init)): complete 2022-03-29T17:51:58.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - Created 3.0 2022-03-29T17:51:58.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,349 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,349 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.910 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,349 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.6 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:58.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.913 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001146 1 2022-03-29T17:51:58.348980+0000 2022-03-29T17:51:58.914 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3488925, 1648576318.3489804 spent on 1 events 2022-03-29T17:51:58.914 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.914 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.914 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.915 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.915 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.915 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000045 0 0.000000 2022-03-29T17:51:58.915 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - Exiting state: Start, entered at 1648576318.3501332, 0.0 spent on 0 events 2022-03-29T17:51:58.915 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.916 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.916 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.916 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.916 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.917 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,350 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:58.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.919 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:58.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:58.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.959533691s + 0.000000000s) 2022-03-29T17:51:58.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - pg_advance_map(id=28, detail=PGAdvanceMap(pg=3.f from=14 to=14 do_init)): start 2022-03-29T17:51:58.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.c 2022-03-29T17:51:58.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=? pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,349 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.6 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:58.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=? pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:58.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.d 2022-03-29T17:51:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:58.927 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,351 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=0 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.6 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:58.929 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.929 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.929 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - Created 3.11 2022-03-29T17:51:58.931 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,351 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.931 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,351 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:51:58.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,351 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.14 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:58.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,350 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - pg_advance_map(id=27, detail=PGAdvanceMap(pg=3.c from=14 to=14 do_init)): complete 2022-03-29T17:51:58.935 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002938 0 0.000000 2022-03-29T17:51:58.935 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3492007, 0.0 spent on 0 events 2022-03-29T17:51:58.935 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.935 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.937 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000126 1 2022-03-29T17:51:58.352296+0000 2022-03-29T17:51:58.937 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3521912, 1648576318.3522966 spent on 1 events 2022-03-29T17:51:58.937 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.937 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.938 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.938 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.938 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.938 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:51:58.939 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - Exiting state: Start, entered at 1648576318.3523846, 0.0 spent on 0 events 2022-03-29T17:51:58.939 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - handle_peering_op on 3.1a from 2 2022-03-29T17:51:58.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1a from 2 notify: (query:14 sent:14 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:58.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=? pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,352 [shard 0] osd - Creating 3.b 2022-03-29T17:51:58.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=? pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.942 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,353 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1d 2022-03-29T17:51:58.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:58.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - peering_event(id=106, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.5 from 1 notify: (query:14 sent:14 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:58.946 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,352 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.946 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.947 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,353 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.947 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.947 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,353 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.947 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.948 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.948 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.948 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.948 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:58.948 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.949 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:58.949 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.949 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.949 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:58.950 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:58.950 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.950 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,353 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:58.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - handle_peering_op on 3.1b from 0 2022-03-29T17:51:58.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1b from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:58.953 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,354 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - peering_event(id=99, detail=PeeringEvent(from=? pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - pg_advance_map(id=17, detail=PGAdvanceMap(pg=3.1d from=14 to=14 do_init)): complete 2022-03-29T17:51:58.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - Creating 3.7 2022-03-29T17:51:58.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,354 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.14 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:58.954 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,354 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.14 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:58.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:58.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - Created 3.f 2022-03-29T17:51:58.956 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,355 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:58.957 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,355 [shard 0] osd - pg_epoch 14 pg[3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.957 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,355 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1a from 2 notify: (query:14 sent:14 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:58.957 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,355 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.960 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,355 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.960 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,355 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.c 2022-03-29T17:51:58.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:58.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:58.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=? pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:58.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=? pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:58.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.17 from 1 notify: (query:14 sent:14 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.17 2022-03-29T17:51:58.964 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,355 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.14 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:58.964 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:58.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:58.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:58.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.11 2022-03-29T17:51:58.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=? pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - peering_event(id=109, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.14 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:58.966 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,356 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:58.967 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:58.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - pg_epoch 14 pg[3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:51:58.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:51:58.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.052658081s + 0.000000000s) 2022-03-29T17:51:58.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - pg_advance_map(id=43, detail=PGAdvanceMap(pg=3.b from=14 to=14 do_init)): start 2022-03-29T17:51:58.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,356 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1a from 2 notify: (query:14 sent:14 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:58.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:58.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:58.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:58.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:58.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:58.975 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.060428 4 2022-03-29T17:51:58.307803+0000 2022-03-29T17:51:58.975 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.296722, 1648576318.3078039 spent on 4 events 2022-03-29T17:51:58.975 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:58.975 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:58.975 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:58.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:58.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:58.976 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:58.976 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:58.977 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:58.977 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:58.977 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:58.977 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:58.978 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:58.978 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:58.978 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:58.978 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:58.979 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000600 0 0.000000 2022-03-29T17:51:58.979 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.3572156, 0.0 spent on 0 events 2022-03-29T17:51:58.979 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:58.979 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:58.979 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:58.980 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000053 0 0.000000 2022-03-29T17:51:58.980 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.3578494, 0.0 spent on 0 events 2022-03-29T17:51:58.980 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:58.980 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,357 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:58.981 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.981 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.17 from 1 notify: (query:14 sent:14 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:58.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:58.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.0 2022-03-29T17:51:58.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - log is not dirty 2022-03-29T17:51:58.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=? pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:58.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_advance_map(id=42, detail=PGAdvanceMap(pg=3.0 from=14 to=14 do_init)): complete 2022-03-29T17:51:58.985 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003115 0 0.000000 2022-03-29T17:51:58.985 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,358 [shard 0] osd - Exiting state: Initial, entered at 1648576318.355213, 0.0 spent on 0 events 2022-03-29T17:51:58.985 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:58.985 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,358 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:58.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:58.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:58.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:58.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:58.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:58.988 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,358 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:58.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - peering_event(id=108, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:58.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:58.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:58.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:51:58.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:58.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:51:58.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.991905212s + 0.000000000s) 2022-03-29T17:51:58.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,358 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:58.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,357 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:58.993 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000112 1 2022-03-29T17:51:58.358460+0000 2022-03-29T17:51:58.993 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3583696, 1648576318.3584604 spent on 1 events 2022-03-29T17:51:58.993 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:58.993 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - Entering state: Started 2022-03-29T17:51:58.994 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:58.994 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - Entering state: Start 2022-03-29T17:51:58.994 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:58.994 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000045 0 0.000000 2022-03-29T17:51:58.995 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - Exiting state: Start, entered at 1648576318.3591866, 0.0 spent on 0 events 2022-03-29T17:51:58.995 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:58.995 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:58.995 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:58.996 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:58.996 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.996 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,359 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:58.996 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:58.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:58.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:58.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:58.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:51:58.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:58.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - Creating 3.3 2022-03-29T17:51:58.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.001 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_advance_map(id=19, detail=PGAdvanceMap(pg=3.7 from=14 to=14 do_init)): start 2022-03-29T17:51:59.001 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_advance_map(id=18, detail=PGAdvanceMap(pg=3.11 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.001 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002130 0 0.000000 2022-03-29T17:51:59.002 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3580453, 0.0 spent on 0 events 2022-03-29T17:51:59.002 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.002 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.002 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.002 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.003 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000125 1 2022-03-29T17:51:58.360327+0000 2022-03-29T17:51:59.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3602228, 1648576318.3603272 spent on 1 events 2022-03-29T17:51:59.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:51:59.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - Exiting state: Start, entered at 1648576318.3604095, 0.0 spent on 0 events 2022-03-29T17:51:59.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,359 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - Created 3.b 2022-03-29T17:51:59.010 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.010 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,360 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.011 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1b from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.011 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,360 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=? pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - handle_peering_op on 3.10 from 1 2022-03-29T17:51:59.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.10 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.013 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.013 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.013 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.016 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.016 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.017 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.017 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.017 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.017 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.018 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.018 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:59.018 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.018 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.018 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:59.019 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.019 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.019 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.019 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.020 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.020 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.020 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.020 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.020 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.021 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.021 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - Created 3.7 2022-03-29T17:51:59.021 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,361 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.021 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,361 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.0 2022-03-29T17:51:59.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.024 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,362 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1b from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1b from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - pg_epoch 14 pg[3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 2} acting {0, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 2}+{0, 2} 2022-03-29T17:51:59.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000145510} 2022-03-29T17:51:59.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.971557617s + 0.000000000s) 2022-03-29T17:51:59.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - pg_advance_map(id=29, detail=PGAdvanceMap(pg=3.3 from=14 to=14 do_init)): start 2022-03-29T17:51:59.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.17 from 1 notify: (query:14 sent:14 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.028 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,362 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,362 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.030 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,363 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.031 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.031 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,363 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.031 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.031 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=? pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.031 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - peering_event(id=96, detail=PeeringEvent(from=? pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.032 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - Creating 3.18 2022-03-29T17:51:59.032 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.11 2022-03-29T17:51:59.032 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.032 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.033 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.033 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.033 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - peering_event(id=302, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - peering_event(id=302, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - Creating 3.4 2022-03-29T17:51:59.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:51:59.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=0 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1b from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,363 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1a from 2 notify: (query:14 sent:14 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.1a 2022-03-29T17:51:59.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,364 [shard 0] osd - pg_advance_map(id=28, detail=PGAdvanceMap(pg=3.f from=14 to=14 do_init)): complete 2022-03-29T17:51:59.039 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002791 0 0.000000 2022-03-29T17:51:59.039 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3613632, 0.0 spent on 0 events 2022-03-29T17:51:59.039 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.039 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.039 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,364 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.041 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000135 1 2022-03-29T17:51:58.364327+0000 2022-03-29T17:51:59.041 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3642154, 1648576318.3643272 spent on 1 events 2022-03-29T17:51:59.041 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.041 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.042 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.042 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.042 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.042 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:51:59.042 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - Exiting state: Start, entered at 1648576318.3644133, 0.0 spent on 0 events 2022-03-29T17:51:59.043 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,364 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.043 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,364 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.047 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.065817 4 2022-03-29T17:51:58.313714+0000 2022-03-29T17:51:59.047 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.2993581, 1648576318.3137147 spent on 4 events 2022-03-29T17:51:59.047 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.047 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.048 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.048 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.049 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.049 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.2 (up) accepted 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.049 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 2} async_recovery= 2022-03-29T17:51:59.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.053 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.053 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.053 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.053 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.054 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.054 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,365 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.054 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.054 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,2 down blocked_by {} 2022-03-29T17:51:59.055 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.055 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.055 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:59.055 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,365 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.058 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,364 [shard 0] osd - pg_epoch 14 pg[3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.059 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.059 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {2, 0} acting {2, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.059 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.059 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 0}+{2, 0} 2022-03-29T17:51:59.060 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.060 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000123930} 2022-03-29T17:51:59.060 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.999389648s + 0.000000000s) 2022-03-29T17:51:59.060 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.061 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_advance_map(id=20, detail=PGAdvanceMap(pg=3.18 from=14 to=14 do_init)): start 2022-03-29T17:51:59.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.001319 0 0.000000 2022-03-29T17:51:59.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.3652208, 0.0 spent on 0 events 2022-03-29T17:51:59.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000071 0 0.000000 2022-03-29T17:51:59.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.3665822, 0.0 spent on 0 events 2022-03-29T17:51:59.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1a from 2 notify: (query:14 sent:14 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.066 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,366 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,366 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.067 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.068 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.068 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - Created 3.3 2022-03-29T17:51:59.068 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,367 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.068 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.068 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,367 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.f 2022-03-29T17:51:59.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=? pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.070 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.070 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,367 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.071 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.071 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.7 2022-03-29T17:51:59.071 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.072 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.10 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.073 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=? pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.073 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.073 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - handle_peering_op on 3.16 from 1 2022-03-29T17:51:59.074 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.16 from 1 notify: (query:14 sent:14 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.074 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,367 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002943 0 0.000000 2022-03-29T17:51:59.074 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,367 [shard 0] osd - Exiting state: Initial, entered at 1648576318.364954, 0.0 spent on 0 events 2022-03-29T17:51:59.074 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,367 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.075 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,367 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.075 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=? pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,368 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=? pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,368 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.b 2022-03-29T17:51:59.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,368 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,368 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=? pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,368 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.080 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,367 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.080 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.080 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.080 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.081 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.081 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001200 1 2022-03-29T17:51:58.369113+0000 2022-03-29T17:51:59.081 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3679419, 1648576318.3691137 spent on 1 events 2022-03-29T17:51:59.081 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.082 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.082 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.082 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.082 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.082 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000055 0 0.000000 2022-03-29T17:51:59.083 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - Exiting state: Start, entered at 1648576318.3692183, 0.0 spent on 0 events 2022-03-29T17:51:59.083 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.083 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.083 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.084 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.084 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.084 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,369 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.084 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.085 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,2 down blocked_by {} 2022-03-29T17:51:59.085 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.085 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.085 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:51:59.086 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - handle_authentication osd.2 has caps allow * 2022-03-29T17:51:59.086 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - Creating 3.12 2022-03-29T17:51:59.087 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,369 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.10 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.087 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,369 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.10 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.088 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,370 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.088 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,370 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,370 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,370 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.093 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.093 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.093 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.093 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.094 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.094 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.094 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.094 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.095 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.095 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.095 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - Created 3.18 2022-03-29T17:51:59.095 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,370 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.095 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.096 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,370 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.096 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.096 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - peering_event(id=110, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.096 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.7 2022-03-29T17:51:59.097 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.097 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.097 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.097 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.098 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.099 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,370 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.099 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.099 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.f 2022-03-29T17:51:59.100 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.100 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.100 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.100 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.100 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_epoch 14 pg[3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:51:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:51:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.067657471s + 0.000000000s) 2022-03-29T17:51:59.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_advance_map(id=44, detail=PGAdvanceMap(pg=3.4 from=14 to=14 do_init)): start 2022-03-29T17:51:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=? pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - pg_advance_map(id=43, detail=PGAdvanceMap(pg=3.b from=14 to=14 do_init)): complete 2022-03-29T17:51:59.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,371 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.b 2022-03-29T17:51:59.105 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,372 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=? pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.105 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,372 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.106 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,372 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.106 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,372 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.16 from 1 notify: (query:14 sent:14 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.107 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,372 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.10 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.107 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,372 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.107 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,372 [shard 0] osd - pg_epoch 14 pg[3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.107 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,372 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - peering_event(id=302, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1a from 2 notify: (query:14 sent:14 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,373 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003035 0 0.000000 2022-03-29T17:51:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,373 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3703437, 0.0 spent on 0 events 2022-03-29T17:51:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,373 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,373 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - pg_advance_map(id=19, detail=PGAdvanceMap(pg=3.7 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - peering_event(id=93, detail=PeeringEvent(from=? pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.114 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.114 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,373 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.114 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.114 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,373 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.16 from 1 notify: (query:14 sent:14 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.116 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,372 [shard 0] osd - pg_epoch 14 pg[3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.116 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.117 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:59.117 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.117 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:59.117 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.982994080s + 0.000000000s) 2022-03-29T17:51:59.117 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.118 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_advance_map(id=30, detail=PGAdvanceMap(pg=3.12 from=14 to=14 do_init)): start 2022-03-29T17:51:59.118 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.3 2022-03-29T17:51:59.118 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.118 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=? pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.119 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.119 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_advance_map(id=29, detail=PGAdvanceMap(pg=3.3 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.119 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.123 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001150 1 2022-03-29T17:51:58.374556+0000 2022-03-29T17:51:59.123 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3734295, 1648576318.374556 spent on 1 events 2022-03-29T17:51:59.123 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.124 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.124 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.124 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.124 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.125 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000042 0 0.000000 2022-03-29T17:51:59.125 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - Exiting state: Start, entered at 1648576318.3746643, 0.0 spent on 0 events 2022-03-29T17:51:59.125 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.125 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.125 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,374 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:59.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:59.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - peering_event(id=111, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.130 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.130 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=? pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.130 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,375 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002488 0 0.000000 2022-03-29T17:51:59.131 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,375 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3727982, 0.0 spent on 0 events 2022-03-29T17:51:59.131 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,375 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.131 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,375 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.131 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.131 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.132 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.132 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.132 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.132 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,375 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000130 1 2022-03-29T17:51:58.375420+0000 2022-03-29T17:51:59.134 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,375 [shard 0] monc - added challenge on [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.? -@54393] 2022-03-29T17:51:59.134 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.18 2022-03-29T17:51:59.134 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.134 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=? pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.135 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,375 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.135 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,375 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e14 {3.5=14,2022-03-29T17:51:58.272165+0000,3.7=14,2022-03-29T17:51:58.272165+0000,3.8=14,2022-03-29T17:51:58.272165+0000,3.e=14,2022-03-29T17:51:58.272165+0000,3.11=14,2022-03-29T17:51:58.272165+0000,3.16=14,2022-03-29T17:51:58.272165+0000,3.18=14,2022-03-29T17:51:58.272165+0000,3.1d=14,2022-03-29T17:51:58.272165+0000,3.1e=14,2022-03-29T17:51:58.272165+0000}) v2): sub events complete 2022-03-29T17:51:59.135 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,375 [shard 0] osd - compound_peering_request(id=2, detail=pg_create2(e14 {3.5=14,2022-03-29T17:51:58.272165+0000,3.7=14,2022-03-29T17:51:58.272165+0000,3.8=14,2022-03-29T17:51:58.272165+0000,3.e=14,2022-03-29T17:51:58.272165+0000,3.11=14,2022-03-29T17:51:58.272165+0000,3.16=14,2022-03-29T17:51:58.272165+0000,3.18=14,2022-03-29T17:51:58.272165+0000,3.1d=14,2022-03-29T17:51:58.272165+0000,3.1e=14,2022-03-29T17:51:58.272165+0000}) v2): complete 2022-03-29T17:51:59.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,374 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - Created 3.4 2022-03-29T17:51:59.139 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.140 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.140 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.141 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,376 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.143 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3753295, 1648576318.3754203 spent on 1 events 2022-03-29T17:51:59.143 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.143 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.143 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.144 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.144 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.144 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000065 0 0.000000 2022-03-29T17:51:59.144 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - Exiting state: Start, entered at 1648576318.376563, 0.0 spent on 0 events 2022-03-29T17:51:59.144 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.145 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.145 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.145 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.145 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.146 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,376 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.146 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - pg_advance_map(id=20, detail=PGAdvanceMap(pg=3.18 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,376 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=? pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:59.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:59.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.151 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,377 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.151 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - peering_event(id=302, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - peering_event(id=304, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - handle_peering_op on 3.1f from 0 2022-03-29T17:51:59.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,377 [shard 0] osd - Creating 3.13 2022-03-29T17:51:59.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - Created 3.12 2022-03-29T17:51:59.157 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,378 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.157 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,378 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.158 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,378 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=? pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.160 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:51:59.160 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.2 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.160 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.18 2022-03-29T17:51:59.160 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.161 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.161 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.161 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.161 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.162 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - peering_event(id=98, detail=PeeringEvent(from=? pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.162 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.16 from 1 notify: (query:14 sent:14 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.16 2022-03-29T17:51:59.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,378 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=? pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - Creating 3.a 2022-03-29T17:51:59.163 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,379 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.3 2022-03-29T17:51:59.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - peering_event(id=304, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.166 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.166 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.166 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.167 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.167 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,379 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.167 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,379 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.072848 4 2022-03-29T17:51:58.316831+0000 2022-03-29T17:51:59.167 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,379 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3070374, 1648576318.3168318 spent on 4 events 2022-03-29T17:51:59.168 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,379 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.168 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,379 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,380 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:51:59.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:51:59.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.076332092s + 0.000000000s) 2022-03-29T17:51:59.171 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.172 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.172 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:59.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.174 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.174 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.174 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,380 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.176 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.176 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:59.177 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.177 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.177 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.1 (up) accepted 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.177 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:59.178 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.178 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 1} async_recovery= 2022-03-29T17:51:59.178 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.178 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.179 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.179 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.001282 0 0.000000 2022-03-29T17:51:59.179 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,381 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.379936, 0.0 spent on 0 events 2022-03-29T17:51:59.179 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.180 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,381 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.180 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.180 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000052 0 0.000000 2022-03-29T17:51:59.180 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,381 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.381269, 0.0 spent on 0 events 2022-03-29T17:51:59.180 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,381 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.181 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,381 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.181 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.181 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.16 from 1 notify: (query:14 sent:14 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.181 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - pg_advance_map(id=45, detail=PGAdvanceMap(pg=3.13 from=14 to=14 do_init)): start 2022-03-29T17:51:59.183 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.12 2022-03-29T17:51:59.183 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.184 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=? pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.184 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,381 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.186 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002021 0 0.000000 2022-03-29T17:51:59.186 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3801126, 0.0 spent on 0 events 2022-03-29T17:51:59.187 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.187 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.188 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000127 1 2022-03-29T17:51:58.382294+0000 2022-03-29T17:51:59.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3821893, 1648576318.382295 spent on 1 events 2022-03-29T17:51:59.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.190 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.190 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000042 0 0.000000 2022-03-29T17:51:59.190 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - Exiting state: Start, entered at 1648576318.3823822, 0.0 spent on 0 events 2022-03-29T17:51:59.190 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.191 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.192 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.2 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.192 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.192 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.193 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - peering_event(id=113, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.193 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,382 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.193 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.193 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,382 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.2 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.194 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.2 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.194 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,382 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.195 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.196 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,383 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:59.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.197 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:59.197 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 124.992187500s + 0.000000000s) 2022-03-29T17:51:59.197 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.197 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_advance_map(id=31, detail=PGAdvanceMap(pg=3.a from=14 to=14 do_init)): start 2022-03-29T17:51:59.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,383 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.201 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,383 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:59.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:51:59.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,383 [shard 0] osd - Created 3.13 2022-03-29T17:51:59.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - peering_event(id=114, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.2 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - handle_authentication osd.0 has caps allow * 2022-03-29T17:51:59.208 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.208 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.208 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.209 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001669 0 0.000000 2022-03-29T17:51:59.209 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - Exiting state: Initial, entered at 1648576318.382997, 0.0 spent on 0 events 2022-03-29T17:51:59.209 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.209 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.210 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.210 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.210 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.210 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.211 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.211 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000114 1 2022-03-29T17:51:58.384806+0000 2022-03-29T17:51:59.211 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3847141, 1648576318.3848066 spent on 1 events 2022-03-29T17:51:59.211 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.211 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.212 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.212 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.212 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.212 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000039 0 0.000000 2022-03-29T17:51:59.213 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - Exiting state: Start, entered at 1648576318.3848932, 0.0 spent on 0 events 2022-03-29T17:51:59.213 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,384 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.214 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,385 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,385 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.4 2022-03-29T17:51:59.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=? pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=? pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - pg_advance_map(id=44, detail=PGAdvanceMap(pg=3.4 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.217 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,385 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.218 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,385 [shard 0] osd - peering_event(id=112, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.16 from 1 notify: (query:14 sent:14 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.220 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,386 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.221 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.221 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,386 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.221 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.221 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,386 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:59.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:59.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - Created 3.a 2022-03-29T17:51:59.225 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,386 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:51:59.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.8 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.4 2022-03-29T17:51:59.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - peering_event(id=304, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.229 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,387 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:51:59.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,387 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.0 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.230 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,387 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.230 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,387 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.231 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,387 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.231 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,387 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.0 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.232 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,387 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=? pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,386 [shard 0] osd - peering_event(id=307, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,388 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=0 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.233 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,388 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.0 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.233 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,388 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.0 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.234 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,388 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.234 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,388 [shard 0] osd - peering_event(id=115, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.0 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,388 [shard 0] osd - Creating 3.1c 2022-03-29T17:51:59.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,388 [shard 0] osd - peering_event(id=307, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.235 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,388 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,388 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,388 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=? pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.237 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - handle_peering_op on 3.6 from 1 2022-03-29T17:51:59.238 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.6 from 1 notify: (query:14 sent:14 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.238 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,389 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.238 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.238 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.12 2022-03-29T17:51:59.239 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.239 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.239 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.239 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.240 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.240 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - Creating 3.1 2022-03-29T17:51:59.240 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=? pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - handle_peering_op on 3.1e from 1 2022-03-29T17:51:59.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.13 2022-03-29T17:51:59.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=? pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - pg_advance_map(id=45, detail=PGAdvanceMap(pg=3.13 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.243 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,389 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - handle_peering_op on 3.c from 0 2022-03-29T17:51:59.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1e from 1 notify: (query:14 sent:14 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,390 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1e from 1 notify: (query:14 sent:14 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,390 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1e from 1 notify: (query:14 sent:14 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,390 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1e from 1 notify: (query:14 sent:14 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.1e 2022-03-29T17:51:59.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,390 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,389 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.c from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.247 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,390 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.247 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,390 [shard 0] osd - pg_advance_map(id=30, detail=PGAdvanceMap(pg=3.12 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.247 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,390 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.2 v2:172.21.15.138:6802/35579@61625] server_connect: connection race detected (cs=10389299161227899038, e_cs=15047428695210124814, ss=0) and lose to existing [osd.0(cluster) v2:172.21.15.138:6804/35577@54393 >> osd.? v2:172.21.15.138:6802/35579], ask client to wait 2022-03-29T17:51:59.247 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,390 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.248 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,390 [shard 0] osd - pg_epoch 14 pg[3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.248 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,391 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.072982 4 2022-03-29T17:51:58.332099+0000 2022-03-29T17:51:59.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,391 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3182187, 1648576318.3320992 spent on 4 events 2022-03-29T17:51:59.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.251 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,391 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:59.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.253 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:59.253 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.253 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.253 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.1 (up) accepted 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.253 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:59.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - peering_event(id=304, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.255 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.a 2022-03-29T17:51:59.255 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.255 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=? pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.255 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,391 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:51:59.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:51:59.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.088134766s + 0.000000000s) 2022-03-29T17:51:59.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_advance_map(id=46, detail=PGAdvanceMap(pg=3.1c from=14 to=14 do_init)): start 2022-03-29T17:51:59.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 1} async_recovery= 2022-03-29T17:51:59.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.262 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.001393 0 0.000000 2022-03-29T17:51:59.262 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,392 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.391258, 0.0 spent on 0 events 2022-03-29T17:51:59.262 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.263 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,392 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.263 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000066 0 0.000000 2022-03-29T17:51:59.263 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,392 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.39269, 0.0 spent on 0 events 2022-03-29T17:51:59.264 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,392 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.264 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,392 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1e from 1 notify: (query:14 sent:14 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,392 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.266 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.266 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.266 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_advance_map(id=31, detail=PGAdvanceMap(pg=3.a from=14 to=14 do_init)): complete 2022-03-29T17:51:59.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 2} acting {0, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 2}+{0, 2} 2022-03-29T17:51:59.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000145510} 2022-03-29T17:51:59.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.002311707s + 0.000000000s) 2022-03-29T17:51:59.271 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002881 0 0.000000 2022-03-29T17:51:59.271 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3906925, 0.0 spent on 0 events 2022-03-29T17:51:59.271 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.271 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.273 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000111 1 2022-03-29T17:51:58.393729+0000 2022-03-29T17:51:59.273 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3936398, 1648576318.3937294 spent on 1 events 2022-03-29T17:51:59.273 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.273 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.274 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.274 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.274 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.274 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:51:59.275 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Exiting state: Start, entered at 1648576318.3938148, 0.0 spent on 0 events 2022-03-29T17:51:59.275 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.275 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.275 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.276 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.276 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.276 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,393 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,394 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:51:59.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.4 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.278 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,394 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@61625 >> osd.? v2:172.21.15.138:6804/35577] execute_connecting(): going to WAIT 2022-03-29T17:51:59.279 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,394 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@61625 >> osd.? v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:51:59.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - peering_event(id=116, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1e from 1 notify: (query:14 sent:14 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.281 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.281 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - pg_advance_map(id=32, detail=PGAdvanceMap(pg=3.1 from=14 to=14 do_init)): start 2022-03-29T17:51:59.281 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,394 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.2 v2:172.21.15.138:6802/35579@61625] execute_accepting(): going to SERVER_WAIT 2022-03-29T17:51:59.281 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,394 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001750 0 0.000000 2022-03-29T17:51:59.282 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,394 [shard 0] osd - Exiting state: Initial, entered at 1648576318.3930516, 0.0 spent on 0 events 2022-03-29T17:51:59.282 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,394 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.282 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,394 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.282 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.282 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,394 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.283 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,394 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000112 1 2022-03-29T17:51:58.394937+0000 2022-03-29T17:51:59.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,393 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:59.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.0 2022-03-29T17:51:59.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.8 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - Created 3.1c 2022-03-29T17:51:59.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,395 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@61625 >> osd.? v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:51:59.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.4 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.291 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,395 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.4 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.4 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,395 [shard 0] osd - peering_event(id=117, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.4 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.294 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - Exiting state: Reset, entered at 1648576318.3948467, 1648576318.394938 spent on 1 events 2022-03-29T17:51:59.294 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.295 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.295 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.295 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.295 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.296 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000049 0 0.000000 2022-03-29T17:51:59.296 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - Exiting state: Start, entered at 1648576318.396054, 0.0 spent on 0 events 2022-03-29T17:51:59.296 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.296 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.297 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.297 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.297 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.297 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,2 down blocked_by {} 2022-03-29T17:51:59.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:59.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.301 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,395 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.301 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,396 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=? pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=? pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.303 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,396 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.8 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.8 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.306 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,396 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.306 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.306 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.306 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.307 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.307 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.307 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - Created 3.1 2022-03-29T17:51:59.307 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,397 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.308 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.308 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,397 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.308 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.308 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,397 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.308 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.309 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,397 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.309 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.309 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.6 from 1 notify: (query:14 sent:14 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.309 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.310 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,397 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.2 v2:172.21.15.138:6802/35579@61625] execute_server_wait(): fault at SERVER_WAIT, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:51:59.310 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,397 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577 >> osd.2 v2:172.21.15.138:6802/35579@61625] closing: reset no, replace no 2022-03-29T17:51:59.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.8 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - Creating 3.19 2022-03-29T17:51:59.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.c from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.13 2022-03-29T17:51:59.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - peering_event(id=307, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,397 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - peering_event(id=309, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.315 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=? pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.315 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=? pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.315 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - Creating 3.15 2022-03-29T17:51:59.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.a 2022-03-29T17:51:59.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.317 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - peering_event(id=309, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.1c 2022-03-29T17:51:59.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,398 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=? pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.319 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,399 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.c from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - handle_peering_op on 3.8 from 1 2022-03-29T17:51:59.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.8 from 1 notify: (query:14 sent:14 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.8 from 1 notify: (query:14 sent:14 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.8 from 1 notify: (query:14 sent:14 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.8 from 1 notify: (query:14 sent:14 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.8 2022-03-29T17:51:59.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,399 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.326 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,399 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.072316 4 2022-03-29T17:51:58.341974+0000 2022-03-29T17:51:59.326 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,399 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3276293, 1648576318.3419743 spent on 4 events 2022-03-29T17:51:59.326 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.326 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:59.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.328 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:59.328 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.328 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.328 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.1 (up) accepted 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.328 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:59.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 1} async_recovery= 2022-03-29T17:51:59.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.330 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000359 0 0.000000 2022-03-29T17:51:59.330 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.400005, 0.0 spent on 0 events 2022-03-29T17:51:59.330 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.331 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.331 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000051 0 0.000000 2022-03-29T17:51:59.331 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4003987, 0.0 spent on 0 events 2022-03-29T17:51:59.332 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.332 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.8 from 1 notify: (query:14 sent:14 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.c from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,400 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=0 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.c from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.334 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,400 [shard 0] osd - pg_epoch 14 pg[3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.335 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,401 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.335 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,401 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.335 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,401 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - pg_epoch 14 pg[3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:51:59.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:51:59.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.098297119s + 0.000000000s) 2022-03-29T17:51:59.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - pg_advance_map(id=47, detail=PGAdvanceMap(pg=3.19 from=14 to=14 do_init)): start 2022-03-29T17:51:59.339 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.6 from 1 notify: (query:14 sent:14 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.339 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - handle_peering_op on 3.d from 1 2022-03-29T17:51:59.340 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,402 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.d from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.340 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,403 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,403 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,403 [shard 0] osd - peering_event(id=118, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.8 from 1 notify: (query:14 sent:14 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,403 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1c 2022-03-29T17:51:59.343 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,403 [shard 0] osd - pg_epoch 14 pg[3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:51:59.343 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,403 [shard 0] osd - Entering state: Initial 2022-03-29T17:51:59.343 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,403 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.343 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,403 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.344 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,403 [shard 0] osd - pg_epoch 14 pg[3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 0 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.344 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,403 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:51:59.344 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:51:59.344 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:51:59.345 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:51:59.345 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.012916565s + 0.000000000s) 2022-03-29T17:51:59.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - peering_event(id=309, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.349 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_advance_map(id=46, detail=PGAdvanceMap(pg=3.1c from=14 to=14 do_init)): complete 2022-03-29T17:51:59.349 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003341 0 0.000000 2022-03-29T17:51:59.350 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - Exiting state: Initial, entered at 1648576318.400975, 0.0 spent on 0 events 2022-03-29T17:51:59.350 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.350 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.351 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000117 1 2022-03-29T17:51:58.404455+0000 2022-03-29T17:51:59.352 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - Exiting state: Reset, entered at 1648576318.404362, 1648576318.4044552 spent on 1 events 2022-03-29T17:51:59.352 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.352 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.352 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.353 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.353 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:51:59.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_advance_map(id=33, detail=PGAdvanceMap(pg=3.15 from=14 to=14 do_init)): start 2022-03-29T17:51:59.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_advance_map(id=32, detail=PGAdvanceMap(pg=3.1 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.6 from 1 notify: (query:14 sent:14 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.6 2022-03-29T17:51:59.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.357 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.357 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.090359 4 2022-03-29T17:51:58.330250+0000 2022-03-29T17:51:59.357 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3145754, 1648576318.3302505 spent on 4 events 2022-03-29T17:51:59.357 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.358 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,404 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.358 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.358 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.361 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000554 0 0.000000 2022-03-29T17:51:59.361 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - Exiting state: Start, entered at 1648576318.4045484, 0.0 spent on 0 events 2022-03-29T17:51:59.361 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.361 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.362 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.362 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.362 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.362 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 1,2 down blocked_by {} 2022-03-29T17:51:59.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.2 2022-03-29T17:51:59.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.368 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.368 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.369 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.369 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.369 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.369 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:59.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.371 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000804 0 0.000000 2022-03-29T17:51:59.371 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4049797, 0.0 spent on 0 events 2022-03-29T17:51:59.371 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.372 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.372 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.372 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000051 0 0.000000 2022-03-29T17:51:59.372 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4058208, 0.0 spent on 0 events 2022-03-29T17:51:59.372 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.373 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,405 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.6 from 1 notify: (query:14 sent:14 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.1 2022-03-29T17:51:59.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,405 [shard 0] osd - Created 3.19 2022-03-29T17:51:59.376 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.376 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=? pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=? pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - peering_event(id=307, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.380 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=? pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.381 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002982 0 0.000000 2022-03-29T17:51:59.381 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - Exiting state: Initial, entered at 1648576318.403735, 0.0 spent on 0 events 2022-03-29T17:51:59.382 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:51:59.382 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - Entering state: Reset 2022-03-29T17:51:59.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:51:59.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:51:59.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:51:59.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:51:59.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:51:59.383 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000127 1 2022-03-29T17:51:58.406871+0000 2022-03-29T17:51:59.383 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - Exiting state: Reset, entered at 1648576318.4067655, 1648576318.4068718 spent on 1 events 2022-03-29T17:51:59.384 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:51:59.384 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - Entering state: Started 2022-03-29T17:51:59.384 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:51:59.384 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,406 [shard 0] osd - Entering state: Start 2022-03-29T17:51:59.385 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:51:59.385 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000052 0 0.000000 2022-03-29T17:51:59.385 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - Exiting state: Start, entered at 1648576318.4069593, 0.0 spent on 0 events 2022-03-29T17:51:59.385 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:51:59.385 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:51:59.386 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering 2022-03-29T17:51:59.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - handle_peering_op on 3.14 from 2 2022-03-29T17:51:59.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.14 from 2 notify: (query:14 sent:14 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - handle_peering_op on 3.10 from 0 2022-03-29T17:51:59.389 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:51:59.389 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.389 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,407 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:51:59.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe 2022-03-29T17:51:59.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} 2022-03-29T17:51:59.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 11 < same_since 14, must notify monitor 2022-03-29T17:51:59.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: no prior_set down osds, will clear prior_readable_until_ub before activating 2022-03-29T17:51:59.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: querying info from osd.1 2022-03-29T17:51:59.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 0:0 2022-03-29T17:51:59.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.d from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,407 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.10 from 0 notify: (query:14 sent:14 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.19 2022-03-29T17:51:59.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=? pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - peering_event(id=312, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - peering_event(id=312, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.395 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,408 [shard 0] osd - compound_peering_request(id=4, detail=pg_create2(e14 {3.0=14,2022-03-29T17:51:58.272165+0000,3.2=14,2022-03-29T17:51:58.272165+0000,3.4=14,2022-03-29T17:51:58.272165+0000,3.b=14,2022-03-29T17:51:58.272165+0000,3.d=14,2022-03-29T17:51:58.272165+0000,3.10=14,2022-03-29T17:51:58.272165+0000,3.13=14,2022-03-29T17:51:58.272165+0000,3.14=14,2022-03-29T17:51:58.272165+0000,3.19=14,2022-03-29T17:51:58.272165+0000,3.1a=14,2022-03-29T17:51:58.272165+0000,3.1c=14,2022-03-29T17:51:58.272165+0000}) v2): sub events complete 2022-03-29T17:51:59.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:1 2022-03-29T17:51:59.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:51:59.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,408 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - Created 3.15 2022-03-29T17:51:59.398 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,409 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering 2022-03-29T17:51:59.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.398 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,409 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.399 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,409 [shard 0] osd - compound_peering_request(id=4, detail=pg_create2(e14 {3.0=14,2022-03-29T17:51:58.272165+0000,3.2=14,2022-03-29T17:51:58.272165+0000,3.4=14,2022-03-29T17:51:58.272165+0000,3.b=14,2022-03-29T17:51:58.272165+0000,3.d=14,2022-03-29T17:51:58.272165+0000,3.10=14,2022-03-29T17:51:58.272165+0000,3.13=14,2022-03-29T17:51:58.272165+0000,3.14=14,2022-03-29T17:51:58.272165+0000,3.19=14,2022-03-29T17:51:58.272165+0000,3.1a=14,2022-03-29T17:51:58.272165+0000,3.1c=14,2022-03-29T17:51:58.272165+0000}) v2): complete 2022-03-29T17:51:59.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - pg_advance_map(id=47, detail=PGAdvanceMap(pg=3.19 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.14 from 2 notify: (query:14 sent:14 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.400 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,409 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=? pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): pg present 2022-03-29T17:51:59.401 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,409 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.d from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.d from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.d from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.403 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,410 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577@54393] server_connect: connection race detected (cs=15047428695210124814, e_cs=10389299161227899038, ss=0) and win, reusing existing [osd.2(cluster) v2:172.21.15.138:6802/35579@61625 >> osd.? v2:172.21.15.138:6804/35577] 2022-03-29T17:51:59.403 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,410 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] closing: reset no, replace no 2022-03-29T17:51:59.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,410 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_accepting(): fault at CLOSING, going to CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:51:59.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,410 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@61625 >> osd.? v2:172.21.15.138:6804/35577] execute_wait(): protocol aborted at REPLACING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:51:59.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,410 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@61625 >> osd.? v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:51:59.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,410 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577@54393] replaced (connected): gs=2, pgs=2, cs=0, client_cookie=15047428695210124814, server_cookie=12774837416581397668, in_seq=0, out_seq=0, out_q=3 2022-03-29T17:51:59.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,409 [shard 0] osd - peering_event(id=309, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.14 from 2 notify: (query:14 sent:14 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.406 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,411 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.19 2022-03-29T17:51:59.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - peering_event(id=312, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.409 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1 2022-03-29T17:51:59.410 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.410 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.410 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.410 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.411 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,411 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,411 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:51:59.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.6 from 1 notify: (query:14 sent:14 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:51:59.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=? pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.10 from 0 notify: (query:14 sent:14 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:51:59.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.11 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.10 from 0 notify: (query:14 sent:14 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:51:59.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,412 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.19 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.19 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.415 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,413 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.19 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.19 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - peering_event(id=119, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.19 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.418 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,413 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.418 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.418 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=? pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.418 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,413 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.419 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.419 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.419 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 NullEvt +create_info for pg: 3.15 2022-03-29T17:51:59.419 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.419 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=? pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:51:59.420 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,413 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.420 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,414 [shard 0] ms - [osd.0(cluster) v2:172.21.15.138:6804/35577@54393 >> osd.2 v2:172.21.15.138:6802/35579] connected: gs=2, pgs=3, cs=0, client_cookie=15047428695210124814, server_cookie=12774837416581397668, in_seq=0, out_seq=0, out_q=3 2022-03-29T17:51:59.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.14 from 2 notify: (query:14 sent:14 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.14 2022-03-29T17:51:59.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.423 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.107379 4 2022-03-29T17:51:58.323317+0000 2022-03-29T17:51:59.424 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,414 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3069324, 1648576318.3233173 spent on 4 events 2022-03-29T17:51:59.424 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.424 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,414 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.425 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.425 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.2 (up) accepted 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 2} async_recovery= 2022-03-29T17:51:59.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.428 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,414 [shard 0] osd - compound_peering_request(id=3, detail=pg_create2(e14 {3.1=14,2022-03-29T17:51:58.272165+0000,3.3=14,2022-03-29T17:51:58.272165+0000,3.6=14,2022-03-29T17:51:58.272165+0000,3.9=14,2022-03-29T17:51:58.272165+0000,3.a=14,2022-03-29T17:51:58.272165+0000,3.c=14,2022-03-29T17:51:58.272165+0000,3.f=14,2022-03-29T17:51:58.272165+0000,3.12=14,2022-03-29T17:51:58.272165+0000,3.15=14,2022-03-29T17:51:58.272165+0000,3.17=14,2022-03-29T17:51:58.272165+0000,3.1b=14,2022-03-29T17:51:58.272165+0000,3.1f=14,2022-03-29T17:51:58.272165+0000}) v2): sub events complete 2022-03-29T17:51:59.429 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,414 [shard 0] osd - compound_peering_request(id=3, detail=pg_create2(e14 {3.1=14,2022-03-29T17:51:58.272165+0000,3.3=14,2022-03-29T17:51:58.272165+0000,3.6=14,2022-03-29T17:51:58.272165+0000,3.9=14,2022-03-29T17:51:58.272165+0000,3.a=14,2022-03-29T17:51:58.272165+0000,3.c=14,2022-03-29T17:51:58.272165+0000,3.f=14,2022-03-29T17:51:58.272165+0000,3.12=14,2022-03-29T17:51:58.272165+0000,3.15=14,2022-03-29T17:51:58.272165+0000,3.17=14,2022-03-29T17:51:58.272165+0000,3.1b=14,2022-03-29T17:51:58.272165+0000,3.1f=14,2022-03-29T17:51:58.272165+0000}) v2): complete 2022-03-29T17:51:59.429 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.15 2022-03-29T17:51:59.429 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Ending blocked outgoing recovery messages 2022-03-29T17:51:59.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:51:59.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.431 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,414 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.431 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,415 [shard 0] osd - handle_peering_op on 3.1b from 1 2022-03-29T17:51:59.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,415 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,415 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000801 0 0.000000 2022-03-29T17:51:59.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4143655, 0.0 spent on 0 events 2022-03-29T17:51:59.434 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.434 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,415 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.434 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000065 0 0.000000 2022-03-29T17:51:59.435 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4152055, 0.0 spent on 0 events 2022-03-29T17:51:59.435 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.435 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,415 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,415 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.14 from 2 notify: (query:14 sent:14 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,415 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.436 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,415 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1b from 1 notify: (query:14 sent:14 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.437 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,415 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.437 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,416 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,416 [shard 0] osd - handle_peering_op on 3.9 from 0 2022-03-29T17:51:59.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,416 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.9 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,416 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.9 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.440 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,416 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.9 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,416 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.9 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,416 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,416 [shard 0] osd - peering_event(id=120, detail=PeeringEvent(from=0 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.9 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,416 [shard 0] osd - handle_peering_op on 3.3 from 0 2022-03-29T17:51:59.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.3 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.3 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.442 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.3 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.3 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=121, detail=PeeringEvent(from=0 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.3 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - handle_peering_op on 3.1 from 0 2022-03-29T17:51:59.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.443 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.444 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.444 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.444 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,417 [shard 0] osd - peering_event(id=122, detail=PeeringEvent(from=0 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - handle_peering_op on 3.f from 0 2022-03-29T17:51:59.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.446 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,418 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.10 from 0 notify: (query:14 sent:14 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.10 2022-03-29T17:51:59.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=? pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 NullEvt +create_info)): complete 2022-03-29T17:51:59.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - pg_advance_map(id=33, detail=PGAdvanceMap(pg=3.15 from=14 to=14 do_init)): complete 2022-03-29T17:51:59.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1b from 1 notify: (query:14 sent:14 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,418 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1b from 1 notify: (query:14 sent:14 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.451 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.451 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.452 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.100596 4 2022-03-29T17:51:58.330216+0000 2022-03-29T17:51:59.452 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3185608, 1648576318.3302164 spent on 4 events 2022-03-29T17:51:59.452 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.452 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:51:59.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.453 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.454 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.454 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.454 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.454 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.455 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.455 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:51:59.455 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.455 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.456 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.456 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000277 0 0.000000 2022-03-29T17:51:59.456 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4192, 0.0 spent on 0 events 2022-03-29T17:51:59.456 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.456 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.457 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.459 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.459 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.459 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.459 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - handle_peering_op on 3.b from 1 2022-03-29T17:51:59.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.b from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1b from 1 notify: (query:14 sent:14 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.1b 2022-03-29T17:51:59.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.461 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.097315 4 2022-03-29T17:51:58.338468+0000 2022-03-29T17:51:59.461 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3226664, 1648576318.3384686 spent on 4 events 2022-03-29T17:51:59.462 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.462 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.462 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.464 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,419 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000044 0 0.000000 2022-03-29T17:51:59.464 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.419529, 0.0 spent on 0 events 2022-03-29T17:51:59.464 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.465 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.465 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.465 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.10 from 0 notify: (query:14 sent:14 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.465 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.11 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - peering_event(id=312, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:51:59.466 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.466 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,420 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.11 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.11 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.470 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.470 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.470 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.470 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.471 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:59.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.472 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.472 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000799 0 0.000000 2022-03-29T17:51:59.472 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4200213, 0.0 spent on 0 events 2022-03-29T17:51:59.472 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.473 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.473 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.473 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000049 0 0.000000 2022-03-29T17:51:59.473 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.420855, 0.0 spent on 0 events 2022-03-29T17:51:59.473 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.474 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,420 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1b from 1 notify: (query:14 sent:14 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,420 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,421 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,421 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.11 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,421 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.14 from 2 notify: (query:14 sent:14 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,421 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.477 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,421 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,421 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,421 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,421 [shard 0] osd - handle_peering_op on 3.11 from 1 2022-03-29T17:51:59.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,421 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.11 from 1 notify: (query:14 sent:14 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.11 from 1 notify: (query:14 sent:14 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.11 from 1 notify: (query:14 sent:14 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.11 from 1 notify: (query:14 sent:14 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.11 2022-03-29T17:51:59.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.483 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.072102 4 2022-03-29T17:51:58.363870+0000 2022-03-29T17:51:59.484 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3502662, 1648576318.3638709 spent on 4 events 2022-03-29T17:51:59.484 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.484 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:59.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.485 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:59.485 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.486 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.486 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.1 (up) accepted 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.486 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:59.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 1} async_recovery= 2022-03-29T17:51:59.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.487 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000293 0 0.000000 2022-03-29T17:51:59.488 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4224107, 0.0 spent on 0 events 2022-03-29T17:51:59.488 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.488 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.488 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.489 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000068 0 0.000000 2022-03-29T17:51:59.489 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4227366, 0.0 spent on 0 events 2022-03-29T17:51:59.489 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.489 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,422 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.489 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.11 from 1 notify: (query:14 sent:14 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,422 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,423 [shard 0] osd - handle_peering_op on 3.e from 2 2022-03-29T17:51:59.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,423 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,423 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.b from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.492 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,423 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,423 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,423 [shard 0] osd - peering_event(id=123, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.11 from 1 notify: (query:14 sent:14 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.494 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=0 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.f from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.494 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,424 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.494 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.494 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.10 from 0 notify: (query:14 sent:14 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - handle_peering_op on 3.12 from 0 2022-03-29T17:51:59.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.12 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:51:59.497 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,424 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.b from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.b from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,424 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.b from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.498 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,425 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1b from 1 notify: (query:14 sent:14 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.499 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.e from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.7 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.7 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.501 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.7 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,425 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.7 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,426 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.12 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,426 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,426 [shard 0] osd - handle_peering_op on 3.1d from 2 2022-03-29T17:51:59.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,426 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1d from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,426 [shard 0] osd - handle_peering_op on 3.e from 0 2022-03-29T17:51:59.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,426 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.e from 0 notify: (query:14 sent:14 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,426 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.e from 0 notify: (query:14 sent:14 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.e from 0 notify: (query:14 sent:14 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.e from 0 notify: (query:14 sent:14 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.e 2022-03-29T17:51:59.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.507 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.128470 4 2022-03-29T17:51:58.310193+0000 2022-03-29T17:51:59.508 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,427 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.2988102, 1648576318.3101933 spent on 4 events 2022-03-29T17:51:59.508 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.508 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,427 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:59.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.509 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:59.509 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.510 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.510 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.0 (up) accepted 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.510 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:59.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 0} async_recovery= 2022-03-29T17:51:59.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,2 2022-03-29T17:51:59.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.7 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.513 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,427 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.12 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.513 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.12 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.513 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.513 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=0 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.12 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.514 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - handle_peering_op on 3.a from 0 2022-03-29T17:51:59.514 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,427 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.a from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.515 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,428 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000260 0 0.000000 2022-03-29T17:51:59.515 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,428 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4273193, 0.0 spent on 0 events 2022-03-29T17:51:59.516 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,428 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.516 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,428 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,428 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.516 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,428 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000051 0 0.000000 2022-03-29T17:51:59.516 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,428 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4281523, 0.0 spent on 0 events 2022-03-29T17:51:59.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,428 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1d from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.518 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,428 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.518 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,428 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,428 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,428 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.e from 0 notify: (query:14 sent:14 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,428 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.520 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,429 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1d from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1d from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1d from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - handle_peering_op on 3.1f from 1 2022-03-29T17:51:59.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1f from 1 notify: (query:14 sent:14 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.a from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.522 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,429 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.a from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.a from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,429 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=0 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.a from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.524 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,430 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,430 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,430 [shard 0] osd - peering_event(id=124, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.e from 0 notify: (query:14 sent:14 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,430 [shard 0] osd - handle_peering_op on 3.18 from 2 2022-03-29T17:51:59.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,430 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.18 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,430 [shard 0] osd - handle_peering_op on 3.2 from 2 2022-03-29T17:51:59.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,430 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.2 from 2 notify: (query:14 sent:14 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,430 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.2 from 2 notify: (query:14 sent:14 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,430 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.2 from 2 notify: (query:14 sent:14 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1f from 1 notify: (query:14 sent:14 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.18 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.528 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,431 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.18 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.18 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1f from 1 notify: (query:14 sent:14 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:51:59.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.18 from 2 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - handle_peering_op on 3.d from 0 2022-03-29T17:51:59.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.d from 0 notify: (query:14 sent:14 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.2 from 2 notify: (query:14 sent:14 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.2 2022-03-29T17:51:59.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,431 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.533 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,431 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.104546 4 2022-03-29T17:51:58.342494+0000 2022-03-29T17:51:59.533 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,431 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3274093, 1648576318.342495 spent on 4 events 2022-03-29T17:51:59.533 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.534 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.536 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1f from 1 notify: (query:14 sent:14 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.1f 2022-03-29T17:51:59.536 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.536 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.536 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.537 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.537 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.099659 4 2022-03-29T17:51:58.347921+0000 2022-03-29T17:51:59.537 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3327513, 1648576318.3479214 spent on 4 events 2022-03-29T17:51:59.538 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.538 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.538 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.539 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.539 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.539 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.539 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.540 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.540 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.542 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.542 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.543 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.543 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.2 (up) accepted 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.543 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 2} async_recovery= 2022-03-29T17:51:59.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000773 0 0.000000 2022-03-29T17:51:59.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4319954, 0.0 spent on 0 events 2022-03-29T17:51:59.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.545 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.546 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000045 0 0.000000 2022-03-29T17:51:59.546 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4328008, 0.0 spent on 0 events 2022-03-29T17:51:59.546 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.546 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,432 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.2 from 2 notify: (query:14 sent:14 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,432 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.549 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:59.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.550 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.551 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000792 0 0.000000 2022-03-29T17:51:59.551 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4324546, 0.0 spent on 0 events 2022-03-29T17:51:59.551 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.551 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000056 0 0.000000 2022-03-29T17:51:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4332833, 0.0 spent on 0 events 2022-03-29T17:51:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.552 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.553 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.553 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1f from 1 notify: (query:14 sent:14 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.553 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.d from 0 notify: (query:14 sent:14 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.554 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,433 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.2 from 2 notify: (query:14 sent:14 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,433 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.d from 0 notify: (query:14 sent:14 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - handle_peering_op on 3.0 from 2 2022-03-29T17:51:59.556 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,434 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.557 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.557 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1f from 1 notify: (query:14 sent:14 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.557 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - handle_peering_op on 3.13 from 1 2022-03-29T17:51:59.557 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.13 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.0 from 2 notify: (query:14 sent:14 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - handle_peering_op on 3.15 from 0 2022-03-29T17:51:59.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.15 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.d from 0 notify: (query:14 sent:14 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.d 2022-03-29T17:51:59.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,434 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.561 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.097486 4 2022-03-29T17:51:58.353193+0000 2022-03-29T17:51:59.562 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.13 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.562 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,435 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.13 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.563 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.13 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.563 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.563 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.13 from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.563 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - handle_peering_op on 3.9 from 2 2022-03-29T17:51:59.563 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.9 from 2 notify: (query:14 sent:14 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.566 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3375757, 1648576318.353193 spent on 4 events 2022-03-29T17:51:59.567 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.567 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:51:59.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.568 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.568 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.568 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.569 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.569 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:51:59.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.570 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000274 0 0.000000 2022-03-29T17:51:59.571 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4356287, 0.0 spent on 0 events 2022-03-29T17:51:59.571 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.571 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.571 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000048 0 0.000000 2022-03-29T17:51:59.572 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,435 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.435934, 0.0 spent on 0 events 2022-03-29T17:51:59.572 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,436 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.572 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,436 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.d from 0 notify: (query:14 sent:14 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.9 from 2 notify: (query:14 sent:14 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - handle_peering_op on 3.c from 1 2022-03-29T17:51:59.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.c from 1 notify: (query:14 sent:14 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.574 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.9 from 2 notify: (query:14 sent:14 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.0 from 2 notify: (query:14 sent:14 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.15 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.576 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:51:58,436 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.15 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.15 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,436 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:51:59.578 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.c from 1 notify: (query:14 sent:14 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.578 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.c from 1 notify: (query:14 sent:14 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.578 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.9 from 2 notify: (query:14 sent:14 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.9 2022-03-29T17:51:59.578 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.579 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.579 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.579 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.579 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.580 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,437 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.133660 4 2022-03-29T17:51:58.311932+0000 2022-03-29T17:51:59.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.0 from 2 notify: (query:14 sent:14 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=0 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.15 from 0 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.582 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,437 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.d from 0 notify: (query:14 sent:14 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,437 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.0 from 2 notify: (query:14 sent:14 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.0 2022-03-29T17:51:59.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.586 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.303901, 1648576318.3119328 spent on 4 events 2022-03-29T17:51:59.586 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.587 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.587 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.587 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.587 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.588 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.588 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.588 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.588 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.2 (up) accepted 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.588 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 2} async_recovery= 2022-03-29T17:51:59.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,2 2022-03-29T17:51:59.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.590 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000277 0 0.000000 2022-03-29T17:51:59.590 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4381309, 0.0 spent on 0 events 2022-03-29T17:51:59.590 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.591 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.591 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000046 0 0.000000 2022-03-29T17:51:59.591 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4384408, 0.0 spent on 0 events 2022-03-29T17:51:59.591 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.592 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.9 from 2 notify: (query:14 sent:14 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.595 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.090274 4 2022-03-29T17:51:58.363741+0000 2022-03-29T17:51:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.348347, 1648576318.3637412 spent on 4 events 2022-03-29T17:51:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.596 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.597 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.598 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.2 (up) accepted 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.598 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 2} async_recovery= 2022-03-29T17:51:59.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.600 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000231 0 0.000000 2022-03-29T17:51:59.601 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.438665, 0.0 spent on 0 events 2022-03-29T17:51:59.601 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.601 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.602 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000047 0 0.000000 2022-03-29T17:51:59.602 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,438 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4389293, 0.0 spent on 0 events 2022-03-29T17:51:59.602 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,439 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.602 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,439 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,439 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,439 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.0 from 2 notify: (query:14 sent:14 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,439 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,438 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,439 [shard 0] osd - handle_peering_op on 3.1c from 1 2022-03-29T17:51:59.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,439 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1c from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): start 2022-03-29T17:51:59.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,439 [shard 0] osd - handle_peering_op on 3.4 from 2 2022-03-29T17:51:59.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,439 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.4 from 2 notify: (query:14 sent:14 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.606 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,440 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.c from 1 notify: (query:14 sent:14 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.c 2022-03-29T17:51:59.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.610 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.097148 4 2022-03-29T17:51:58.357271+0000 2022-03-29T17:51:59.610 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,440 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3431811, 1648576318.3572717 spent on 4 events 2022-03-29T17:51:59.611 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.611 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,440 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.612 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.612 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.612 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.612 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.613 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:59.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.0 from 2 notify: (query:14 sent:14 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - handle_peering_op on 3.b from 0 2022-03-29T17:51:59.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,440 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.b from 0 notify: (query:14 sent:14 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.4 from 2 notify: (query:14 sent:14 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.619 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000775 0 0.000000 2022-03-29T17:51:59.619 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4403708, 0.0 spent on 0 events 2022-03-29T17:51:59.620 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.620 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.620 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000052 0 0.000000 2022-03-29T17:51:59.621 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.441183, 0.0 spent on 0 events 2022-03-29T17:51:59.621 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.621 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.c from 1 notify: (query:14 sent:14 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1c from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): got map 14 2022-03-29T17:51:59.622 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,441 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - handle_peering_op on 3.3 from 2 2022-03-29T17:51:59.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.4 from 2 notify: (query:14 sent:14 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.b from 0 notify: (query:14 sent:14 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.4 from 2 notify: (query:14 sent:14 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.4 2022-03-29T17:51:59.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,441 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.627 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.067225 4 2022-03-29T17:51:58.387190+0000 2022-03-29T17:51:59.627 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3747873, 1648576318.3871908 spent on 4 events 2022-03-29T17:51:59.628 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.628 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.3 from 2 notify: (query:14 sent:14 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.9 from 2 notify: (query:14 sent:14 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.630 INFO:tasks.ceph.osd.0.smithi138.stderr:WARN 2022-03-29 17:51:58,442 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1c from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): pg absent, did not create 2022-03-29T17:51:59.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1c from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): OSDState is active 2022-03-29T17:51:59.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:51:59.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MQuery 3.1c from 1 query_epoch 14 query: query(info 0'0 epoch_sent 14))): complete 2022-03-29T17:51:59.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.633 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.634 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.634 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.634 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.2 (up) accepted 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.634 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 2} async_recovery= 2022-03-29T17:51:59.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.636 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000782 0 0.000000 2022-03-29T17:51:59.636 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4420512, 0.0 spent on 0 events 2022-03-29T17:51:59.636 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.636 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.637 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000048 0 0.000000 2022-03-29T17:51:59.637 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4428694, 0.0 spent on 0 events 2022-03-29T17:51:59.637 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.638 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,442 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.4 from 2 notify: (query:14 sent:14 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,442 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.b from 0 notify: (query:14 sent:14 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.641 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,443 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.641 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.641 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.c from 1 notify: (query:14 sent:14 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.642 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.3 from 2 notify: (query:14 sent:14 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.642 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.3 from 2 notify: (query:14 sent:14 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.642 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.3 from 2 notify: (query:14 sent:14 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.3 2022-03-29T17:51:59.642 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.644 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.078266 4 2022-03-29T17:51:58.383149+0000 2022-03-29T17:51:59.644 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,443 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3655682, 1648576318.3831499 spent on 4 events 2022-03-29T17:51:59.644 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.644 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,443 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.644 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.645 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.645 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.646 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.646 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.2 (up) accepted 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.646 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 2} async_recovery= 2022-03-29T17:51:59.647 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,2 2022-03-29T17:51:59.648 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.4 from 2 notify: (query:14 sent:14 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.b from 0 notify: (query:14 sent:14 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.b 2022-03-29T17:51:59.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.652 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.652 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.652 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000781 0 0.000000 2022-03-29T17:51:59.653 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4438872, 0.0 spent on 0 events 2022-03-29T17:51:59.653 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.653 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.653 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.654 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000050 0 0.000000 2022-03-29T17:51:59.654 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.444703, 0.0 spent on 0 events 2022-03-29T17:51:59.654 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.654 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,444 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.654 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.3 from 2 notify: (query:14 sent:14 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,444 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - handle_peering_op on 3.1 from 2 2022-03-29T17:51:59.655 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1 from 2 notify: (query:14 sent:14 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.658 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.085255 4 2022-03-29T17:51:58.375126+0000 2022-03-29T17:51:59.658 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3593173, 1648576318.3751266 spent on 4 events 2022-03-29T17:51:59.659 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.659 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:51:59.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.660 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.660 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.660 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.661 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.661 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:51:59.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.662 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000281 0 0.000000 2022-03-29T17:51:59.663 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4451406, 0.0 spent on 0 events 2022-03-29T17:51:59.663 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.663 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.663 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000050 0 0.000000 2022-03-29T17:51:59.664 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4454541, 0.0 spent on 0 events 2022-03-29T17:51:59.664 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.664 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.b from 0 notify: (query:14 sent:14 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.665 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - handle_peering_op on 3.f from 1 2022-03-29T17:51:59.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.f from 1 notify: (query:14 sent:14 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.666 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,445 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.666 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,445 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,446 [shard 0] osd - handle_peering_op on 3.19 from 2 2022-03-29T17:51:59.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,446 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.19 from 2 notify: (query:14 sent:14 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,446 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.3 from 2 notify: (query:14 sent:14 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,446 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1 from 2 notify: (query:14 sent:14 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,446 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1 from 2 notify: (query:14 sent:14 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,446 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.f from 1 notify: (query:14 sent:14 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,446 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.f from 1 notify: (query:14 sent:14 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1 from 2 notify: (query:14 sent:14 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.1 2022-03-29T17:51:59.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.673 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,447 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.19 from 2 notify: (query:14 sent:14 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - handle_peering_op on 3.13 from 0 2022-03-29T17:51:59.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.13 from 0 notify: (query:14 sent:14 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.b from 0 notify: (query:14 sent:14 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.19 from 2 notify: (query:14 sent:14 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.13 from 0 notify: (query:14 sent:14 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.13 from 0 notify: (query:14 sent:14 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.19 from 2 notify: (query:14 sent:14 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.19 2022-03-29T17:51:59.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.2 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 2 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.676 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.042728 4 2022-03-29T17:51:58.412461+0000 2022-03-29T17:51:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,447 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.4052138, 1648576318.412461 spent on 4 events 2022-03-29T17:51:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,447 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,447 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.683 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.051992 4 2022-03-29T17:51:58.413733+0000 2022-03-29T17:51:59.683 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3961895, 1648576318.4137337 spent on 4 events 2022-03-29T17:51:59.683 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.684 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.684 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.685 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.685 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.685 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.2 (up) accepted 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.685 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 2} async_recovery= 2022-03-29T17:51:59.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,2 2022-03-29T17:51:59.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.687 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000263 0 0.000000 2022-03-29T17:51:59.687 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4482222, 0.0 spent on 0 events 2022-03-29T17:51:59.687 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.687 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.688 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000048 0 0.000000 2022-03-29T17:51:59.688 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.448517, 0.0 spent on 0 events 2022-03-29T17:51:59.688 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.689 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1 from 2 notify: (query:14 sent:14 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.f from 1 notify: (query:14 sent:14 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.f 2022-03-29T17:51:59.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.691 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.095335 4 2022-03-29T17:51:58.373991+0000 2022-03-29T17:51:59.691 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.353548, 1648576318.3739915 spent on 4 events 2022-03-29T17:51:59.691 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.691 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,448 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,448 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.692 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.693 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.693 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.693 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.693 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.2 (up) accepted 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 2} async_recovery= 2022-03-29T17:51:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 2} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.001337 0 0.000000 2022-03-29T17:51:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4479809, 0.0 spent on 0 events 2022-03-29T17:51:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000049 0 0.000000 2022-03-29T17:51:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4493709, 0.0 spent on 0 events 2022-03-29T17:51:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.19 from 2 notify: (query:14 sent:14 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.703 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:59.703 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.703 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.704 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000808 0 0.000000 2022-03-29T17:51:59.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.448921, 0.0 spent on 0 events 2022-03-29T17:51:59.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.704 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.705 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.705 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000062 0 0.000000 2022-03-29T17:51:59.705 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4497674, 0.0 spent on 0 events 2022-03-29T17:51:59.705 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.706 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,449 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.706 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.706 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.f from 1 notify: (query:14 sent:14 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.706 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,449 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.13 from 0 notify: (query:14 sent:14 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.13 2022-03-29T17:51:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.066798 4 2022-03-29T17:51:58.399423+0000 2022-03-29T17:51:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,450 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3835258, 1648576318.3994231 spent on 4 events 2022-03-29T17:51:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,450 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:51:59.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.712 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:51:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.715 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - handle_peering_op on 3.12 from 1 2022-03-29T17:51:59.715 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.12 from 1 notify: (query:14 sent:14 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.715 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,450 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.716 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.716 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1 from 2 notify: (query:14 sent:14 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.716 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,450 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.716 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.717 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,450 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.f from 1 notify: (query:14 sent:14 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.717 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.12 from 1 notify: (query:14 sent:14 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000785 0 0.000000 2022-03-29T17:51:59.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4503658, 0.0 spent on 0 events 2022-03-29T17:51:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000047 0 0.000000 2022-03-29T17:51:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.451187, 0.0 spent on 0 events 2022-03-29T17:51:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.13 from 0 notify: (query:14 sent:14 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.19 from 2 notify: (query:14 sent:14 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.12 from 1 notify: (query:14 sent:14 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.12 from 1 notify: (query:14 sent:14 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.12 2022-03-29T17:51:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.726 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.075184 4 2022-03-29T17:51:58.391196+0000 2022-03-29T17:51:59.726 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3767335, 1648576318.3911967 spent on 4 events 2022-03-29T17:51:59.726 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.727 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,451 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.728 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.13 from 0 notify: (query:14 sent:14 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - handle_peering_op on 3.1c from 0 2022-03-29T17:51:59.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1c from 0 notify: (query:14 sent:14 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1c from 0 notify: (query:14 sent:14 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.732 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.732 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.732 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.732 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.733 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:59.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.734 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.734 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000805 0 0.000000 2022-03-29T17:51:59.734 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4519567, 0.0 spent on 0 events 2022-03-29T17:51:59.734 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.735 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.735 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000047 0 0.000000 2022-03-29T17:51:59.735 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4527957, 0.0 spent on 0 events 2022-03-29T17:51:59.735 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.736 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,452 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.12 from 1 notify: (query:14 sent:14 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,452 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1c from 0 notify: (query:14 sent:14 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1c from 0 notify: (query:14 sent:14 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.1c 2022-03-29T17:51:59.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.059541 4 2022-03-29T17:51:58.405614+0000 2022-03-29T17:51:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3939264, 1648576318.4056146 spent on 4 events 2022-03-29T17:51:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.1 because it is current primary 2022-03-29T17:51:59.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.1 with 3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:51:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr: osd.0 (up) accepted 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.744 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:51:59.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={1, 0} async_recovery= 2022-03-29T17:51:59.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={1, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.746 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000297 0 0.000000 2022-03-29T17:51:59.746 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.453518, 0.0 spent on 0 events 2022-03-29T17:51:59.746 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.747 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.747 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000045 0 0.000000 2022-03-29T17:51:59.747 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4538474, 0.0 spent on 0 events 2022-03-29T17:51:59.748 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.748 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,453 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1c from 0 notify: (query:14 sent:14 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,453 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - handle_peering_op on 3.a from 1 2022-03-29T17:51:59.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.a from 1 notify: (query:14 sent:14 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.751 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,454 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.12 from 1 notify: (query:14 sent:14 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.a from 1 notify: (query:14 sent:14 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.a from 1 notify: (query:14 sent:14 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.a from 1 notify: (query:14 sent:14 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.a 2022-03-29T17:51:59.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,454 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.754 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,454 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.068867 4 2022-03-29T17:51:58.400004+0000 2022-03-29T17:51:59.754 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,454 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3860548, 1648576318.4000044 spent on 4 events 2022-03-29T17:51:59.755 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,454 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.755 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,454 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.756 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1c from 0 notify: (query:14 sent:14 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.763 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.763 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.763 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.764 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.764 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:59.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.765 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000616 0 0.000000 2022-03-29T17:51:59.765 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.454965, 0.0 spent on 0 events 2022-03-29T17:51:59.766 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.766 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.766 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000053 0 0.000000 2022-03-29T17:51:59.767 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4556265, 0.0 spent on 0 events 2022-03-29T17:51:59.767 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.767 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,455 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.a from 1 notify: (query:14 sent:14 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,455 [shard 0] osd - handle_peering_op on 3.15 from 1 2022-03-29T17:51:59.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.15 from 1 notify: (query:14 sent:14 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.769 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.a from 1 notify: (query:14 sent:14 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.15 from 1 notify: (query:14 sent:14 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.15 from 1 notify: (query:14 sent:14 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.15 from 1 notify: (query:14 sent:14 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.15 2022-03-29T17:51:59.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.772 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.049005 4 2022-03-29T17:51:58.416869+0000 2022-03-29T17:51:59.772 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.4076583, 1648576318.4168692 spent on 4 events 2022-03-29T17:51:59.773 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.773 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.0 with 3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.774 INFO:tasks.ceph.osd.0.smithi138.stderr:up_primary: 0) selected as primary 2022-03-29T17:51:59.774 INFO:tasks.ceph.osd.0.smithi138.stderr:select_replicated_primary primary is osd.0 with 3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.774 INFO:tasks.ceph.osd.0.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.774 INFO:tasks.ceph.osd.0.smithi138.stderr: osd.1 (up) accepted 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.775 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:51:59.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={0, 1} async_recovery= 2022-03-29T17:51:59.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,1 2022-03-29T17:51:59.775 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={0, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.776 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.776 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000227 0 0.000000 2022-03-29T17:51:59.776 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4567008, 0.0 spent on 0 events 2022-03-29T17:51:59.776 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.776 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,456 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.777 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,456 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.777 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,457 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000043 0 0.000000 2022-03-29T17:51:59.777 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,457 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4569602, 0.0 spent on 0 events 2022-03-29T17:51:59.777 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,457 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.778 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,457 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.778 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,457 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.778 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,457 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.15 from 1 notify: (query:14 sent:14 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.778 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,457 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.778 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,457 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,457 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.780 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,457 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.15 from 1 notify: (query:14 sent:14 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,466 [shard 0] osd - handle_peering_op on 3.7 from 1 2022-03-29T17:51:59.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,466 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.7 from 1 notify: (query:14 sent:14 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,466 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.7 from 1 notify: (query:14 sent:14 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,466 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.7 from 1 notify: (query:14 sent:14 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.7 from 1 notify: (query:14 sent:14 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.7 2022-03-29T17:51:59.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.1 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 1 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.105642 4 2022-03-29T17:51:58.372399+0000 2022-03-29T17:51:59.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3615708, 1648576318.3723998 spent on 4 events 2022-03-29T17:51:59.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.1 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:59.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.797 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:59.797 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.797 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.797 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.1 (up) accepted 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.798 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:59.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 1} async_recovery= 2022-03-29T17:51:59.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 1,2 2022-03-29T17:51:59.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 1} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.799 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000297 0 0.000000 2022-03-29T17:51:59.799 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4672809, 0.0 spent on 0 events 2022-03-29T17:51:59.799 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.800 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.800 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000046 0 0.000000 2022-03-29T17:51:59.800 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.467609, 0.0 spent on 0 events 2022-03-29T17:51:59.801 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.801 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,467 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.7 from 1 notify: (query:14 sent:14 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,467 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.802 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,468 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,468 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,468 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.7 from 1 notify: (query:14 sent:14 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - handle_peering_op on 3.1d from 0 2022-03-29T17:51:59.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1d from 0 notify: (query:14 sent:14 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1d from 0 notify: (query:14 sent:14 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1d from 0 notify: (query:14 sent:14 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1d from 0 notify: (query:14 sent:14 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.1d 2022-03-29T17:51:59.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,469 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.805 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.131313 4 2022-03-29T17:51:58.354884+0000 2022-03-29T17:51:59.805 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3387105, 1648576318.3548844 spent on 4 events 2022-03-29T17:51:59.805 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:59.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.807 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:59.807 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.807 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.807 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.0 (up) accepted 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.808 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:59.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 0} async_recovery= 2022-03-29T17:51:59.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,2 2022-03-29T17:51:59.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.809 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000266 0 0.000000 2022-03-29T17:51:59.809 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.4700606, 0.0 spent on 0 events 2022-03-29T17:51:59.810 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.810 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.810 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000046 0 0.000000 2022-03-29T17:51:59.811 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4703572, 0.0 spent on 0 events 2022-03-29T17:51:59.811 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.811 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.811 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1d from 0 notify: (query:14 sent:14 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.812 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.812 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - handle_peering_op on 3.18 from 0 2022-03-29T17:51:59.812 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.18 from 0 notify: (query:14 sent:14 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): start 2022-03-29T17:51:59.812 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,470 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.18 from 0 notify: (query:14 sent:14 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): got map 14 2022-03-29T17:51:59.813 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,470 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.813 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.813 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - peering_event(id=126, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.1d from 0 notify: (query:14 sent:14 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.813 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.18 from 0 notify: (query:14 sent:14 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): pg present 2022-03-29T17:51:59.813 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.18 from 0 notify: (query:14 sent:14 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info for pg: 3.18 2022-03-29T17:51:59.814 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.814 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Adding osd: 0 peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.814 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common peer features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.814 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common acting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.815 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Common upacting features: 3f01cfbb7ffdffff 2022-03-29T17:51:59.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.102022 4 2022-03-29T17:51:58.380153+0000 2022-03-29T17:51:59.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576318.3693547, 1648576318.3801534 spent on 4 events 2022-03-29T17:51:59.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog 2022-03-29T17:51:59.816 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:51:59.816 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.0 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.816 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting all_info osd.2 3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.816 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering calc_acting prefer osd.2 because it is current primary 2022-03-29T17:51:59.817 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering select_replicated_primary newest update on osd.2 with 3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.817 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:51:59.817 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) 2022-03-29T17:51:59.817 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:51:59.817 INFO:tasks.ceph.osd.2.smithi138.stderr: osd.0 (up) accepted 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.818 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:51:59.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:51:59.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_async_recovery_replicated result want={2, 0} async_recovery= 2022-03-29T17:51:59.818 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering acting_recovery_backfill is 0,2 2022-03-29T17:51:59.819 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering choose_acting want={2, 0} backfill_targets= async_recovery_targets= 2022-03-29T17:51:59.819 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: leaving GetLog 2022-03-29T17:51:59.819 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetLog 0.000272 0 0.000000 2022-03-29T17:51:59.819 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576318.471414, 0.0 spent on 0 events 2022-03-29T17:51:59.820 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.820 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:51:59.820 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: still need up_thru update before going active 2022-03-29T17:51:59.820 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetMissing 0.000079 0 0.000000 2022-03-29T17:51:59.820 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576318.4717166, 0.0 spent on 0 events 2022-03-29T17:51:59.821 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.821 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,471 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:51:59.821 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.821 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.18 from 0 notify: (query:14 sent:14 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): submitting ctx 2022-03-29T17:51:59.822 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,471 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.822 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,472 [shard 0] osd - send_alive want=14 up_thru_wanted=14 2022-03-29T17:51:59.822 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,472 [shard 0] osd - send_alive want=14 <= up_thru_wanted=14; skipping 2022-03-29T17:51:59.822 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:58,472 [shard 0] osd - peering_event(id=127, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MNotifyRec 3.18 from 0 notify: (query:14 sent:14 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) ([0,0] all_participants= intervals=)) features: 0x3f01cfbb7ffdffff +create_info)): complete 2022-03-29T17:51:59.823 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:58,796 [shard 0] alienstore - stat 2022-03-29T17:51:59.823 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:58,811 [shard 0] alienstore - stat 2022-03-29T17:51:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:58,814 [shard 0] alienstore - stat 2022-03-29T17:51:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,834 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:51:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,834 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:51:59.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,834 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:51:59.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,834 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 133.526550293s 2022-03-29T17:51:59.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,834 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 133.526550293s (sent 141.530502319s) 2022-03-29T17:51:59.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,834 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,834 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:51:59.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,834 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,835 [shard 0] osd - handle_peering_op on 1.0 from 1 2022-03-29T17:51:59.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,835 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLease epoch 14 from osd.1 pg_lease(ru 133.526550293s ub 141.530502319s int 16.000000000s))): start 2022-03-29T17:51:59.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,835 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLease epoch 14 from osd.1 pg_lease(ru 133.526550293s ub 141.530502319s int 16.000000000s))): got map 14 2022-03-29T17:51:59.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,835 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:51:59.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLease epoch 14 from osd.1 pg_lease(ru 133.526550293s ub 141.530502319s int 16.000000000s))): pg present 2022-03-29T17:51:59.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MLease epoch 14 from osd.1 pg_lease(ru 133.526550293s ub 141.530502319s int 16.000000000s) for pg: 1.0 2022-03-29T17:51:59.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease pg_lease(ru 133.526550293s ub 141.530502319s int 16.000000000s) 2022-03-29T17:51:59.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active no peer_clock_delta_lb -> ruub 141.445175171s 2022-03-29T17:51:59.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active proc_lease readable_until_ub now 141.445175171s 2022-03-29T17:51:59.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLease epoch 14 from osd.1 pg_lease(ru 133.526550293s ub 141.530502319s int 16.000000000s))): submitting ctx 2022-03-29T17:51:59.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - handle_peering_op on 1.0 from 0 2022-03-29T17:51:59.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,836 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=0 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLeaseAck epoch 14 from osd.0 pg_lease_ack(ruub 141.530502319s))): start 2022-03-29T17:51:59.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=0 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLeaseAck epoch 14 from osd.0 pg_lease_ack(ruub 141.530502319s))): got map 14 2022-03-29T17:51:59.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=0 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLeaseAck epoch 14 from osd.0 pg_lease_ack(ruub 141.530502319s))): pg present 2022-03-29T17:51:59.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 MLeaseAck epoch 14 from osd.0 pg_lease_ack(ruub 141.530502319s) for pg: 1.0 2022-03-29T17:51:59.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 141.530502319s 2022-03-29T17:51:59.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 141.530502319s (sent 141.530502319s) 2022-03-29T17:51:59.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=0 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLeaseAck epoch 14 from osd.0 pg_lease_ack(ruub 141.530502319s))): submitting ctx 2022-03-29T17:51:59.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:58,837 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLease epoch 14 from osd.1 pg_lease(ru 133.526550293s ub 141.530502319s int 16.000000000s))): complete 2022-03-29T17:51:59.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:58,838 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=0 pgid=1.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 MLeaseAck epoch 14 from osd.0 pg_lease_ack(ruub 141.530502319s))): complete 2022-03-29T17:51:59.842 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,286 [shard 0] osd - handle_osd_map osd_map(14..15 src has 1..15) v4 2022-03-29T17:51:59.842 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,286 [shard 0] osd - handle_osd_map epochs [14..15], i have 14, src has [1..15] 2022-03-29T17:51:59.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,287 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:51:59.843 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,288 [shard 0] osd - osd.0: committed_osd_maps(15, 15) 2022-03-29T17:51:59.843 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,289 [shard 0] osd - osd.0: map e 15 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:51:59.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=34, detail=PGAdvanceMap(pg=1.0 from=14 to=15)): start 2022-03-29T17:51:59.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=35, detail=PGAdvanceMap(pg=2.2 from=14 to=15)): start 2022-03-29T17:51:59.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=36, detail=PGAdvanceMap(pg=2.4 from=14 to=15)): start 2022-03-29T17:51:59.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=37, detail=PGAdvanceMap(pg=2.5 from=14 to=15)): start 2022-03-29T17:51:59.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=38, detail=PGAdvanceMap(pg=2.6 from=14 to=15)): start 2022-03-29T17:51:59.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=39, detail=PGAdvanceMap(pg=2.7 from=14 to=15)): start 2022-03-29T17:51:59.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=40, detail=PGAdvanceMap(pg=3.1 from=14 to=15)): start 2022-03-29T17:51:59.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=41, detail=PGAdvanceMap(pg=3.3 from=14 to=15)): start 2022-03-29T17:51:59.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=42, detail=PGAdvanceMap(pg=3.6 from=14 to=15)): start 2022-03-29T17:51:59.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=43, detail=PGAdvanceMap(pg=3.9 from=14 to=15)): start 2022-03-29T17:51:59.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=44, detail=PGAdvanceMap(pg=3.a from=14 to=15)): start 2022-03-29T17:51:59.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=45, detail=PGAdvanceMap(pg=3.c from=14 to=15)): start 2022-03-29T17:51:59.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=46, detail=PGAdvanceMap(pg=3.f from=14 to=15)): start 2022-03-29T17:51:59.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=47, detail=PGAdvanceMap(pg=3.12 from=14 to=15)): start 2022-03-29T17:51:59.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=48, detail=PGAdvanceMap(pg=3.15 from=14 to=15)): start 2022-03-29T17:51:59.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=49, detail=PGAdvanceMap(pg=3.17 from=14 to=15)): start 2022-03-29T17:51:59.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=50, detail=PGAdvanceMap(pg=3.1b from=14 to=15)): start 2022-03-29T17:51:59.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,289 [shard 0] osd - pg_advance_map(id=51, detail=PGAdvanceMap(pg=3.1f from=14 to=15)): start 2022-03-29T17:51:59.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:59.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active state: Started advmap 2022-03-29T17:51:59.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active check_recovery_sources no source osds () went down 2022-03-29T17:51:59.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:51:59.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map 2022-03-29T17:51:59.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:51:59.850 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.850 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.850 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:51:59.850 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:59.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:59.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:59.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,290 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:51:59.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:59.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:59.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:59.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,291 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.855 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,291 [shard 0] osd - handle_osd_map osd_map(15..15 src has 1..15) v4 2022-03-29T17:51:59.855 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,291 [shard 0] osd - handle_osd_map epochs [15..15], i have 14, src has [1..15] 2022-03-29T17:51:59.856 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,292 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:51:59.856 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,293 [shard 0] osd - handle_osd_map osd_map(15..15 src has 1..15) v4 2022-03-29T17:51:59.856 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,293 [shard 0] osd - handle_osd_map epochs [15..15], i have 14, src has [1..15] 2022-03-29T17:51:59.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,294 [shard 0] osd - osd.2: committed_osd_maps(15, 15) 2022-03-29T17:51:59.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,294 [shard 0] osd - osd.2: map e 15 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:51:59.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,294 [shard 0] osd - pg_advance_map(id=21, detail=PGAdvanceMap(pg=2.0 from=14 to=15)): start 2022-03-29T17:51:59.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,294 [shard 0] osd - pg_advance_map(id=22, detail=PGAdvanceMap(pg=2.1 from=14 to=15)): start 2022-03-29T17:51:59.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,294 [shard 0] osd - pg_advance_map(id=23, detail=PGAdvanceMap(pg=2.3 from=14 to=15)): start 2022-03-29T17:51:59.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,294 [shard 0] osd - pg_advance_map(id=24, detail=PGAdvanceMap(pg=3.5 from=14 to=15)): start 2022-03-29T17:51:59.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,294 [shard 0] osd - pg_advance_map(id=25, detail=PGAdvanceMap(pg=3.7 from=14 to=15)): start 2022-03-29T17:51:59.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,294 [shard 0] osd - pg_advance_map(id=26, detail=PGAdvanceMap(pg=3.8 from=14 to=15)): start 2022-03-29T17:51:59.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,294 [shard 0] osd - pg_advance_map(id=27, detail=PGAdvanceMap(pg=3.e from=14 to=15)): start 2022-03-29T17:51:59.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_advance_map(id=28, detail=PGAdvanceMap(pg=3.11 from=14 to=15)): start 2022-03-29T17:51:59.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_advance_map(id=29, detail=PGAdvanceMap(pg=3.16 from=14 to=15)): start 2022-03-29T17:51:59.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_advance_map(id=30, detail=PGAdvanceMap(pg=3.18 from=14 to=15)): start 2022-03-29T17:51:59.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_advance_map(id=31, detail=PGAdvanceMap(pg=3.1d from=14 to=15)): start 2022-03-29T17:51:59.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_advance_map(id=32, detail=PGAdvanceMap(pg=3.1e from=14 to=15)): start 2022-03-29T17:51:59.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:51:59.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,295 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:51:59.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:51:59.877 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.895898 2 0.000837 2022-03-29T17:51:59.877 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,296 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4004838, 0.000837186 spent on 2 events 2022-03-29T17:51:59.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:51:59.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.968885 0 0.000000 2022-03-29T17:51:59.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,296 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3275955, 0.0 spent on 0 events 2022-03-29T17:51:59.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:51:59.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,296 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:51:59.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:51:59.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:51:59.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:51:59.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.929672241s) 2022-03-29T17:51:59.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:51:59.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:51:59.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,296 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:51:59.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:51:59.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:51:59.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:51:59.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:51:59.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:51:59.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:51:59.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:51:59.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:51:59.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:51:59.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:51:59.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:51:59.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:51:59.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:51:59.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:51:59.884 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:51:59.884 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,297 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:51:59.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:51:59.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:51:59.886 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.827466 2 0.000636 2022-03-29T17:51:59.886 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,297 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4704313, 0.000636602 spent on 2 events 2022-03-29T17:51:59.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:51:59.887 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,297 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.959290 0 0.000000 2022-03-29T17:51:59.887 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,298 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.33869, 0.0 spent on 0 events 2022-03-29T17:51:59.887 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:51:59.887 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,298 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:51:59.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:51:59.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:51:59.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:51:59.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.931152344s) 2022-03-29T17:51:59.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:51:59.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:51:59.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:51:59.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:51:59.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:51:59.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:51:59.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:51:59.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:51:59.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:51:59.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:51:59.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:51:59.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:51:59.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:51:59.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:51:59.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:51:59.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:51:59.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:51:59.893 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:51:59.893 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,298 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:51:59.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:51:59.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,298 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:51:59.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:51:59.898 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.831899 2 0.000763 2022-03-29T17:51:59.898 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,299 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4676824, 0.000763727 spent on 2 events 2022-03-29T17:51:59.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:51:59.898 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.938111 0 0.000000 2022-03-29T17:51:59.899 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,299 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3615403, 0.0 spent on 0 events 2022-03-29T17:51:59.899 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:51:59.899 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,299 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:51:59.899 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:51:59.900 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,299 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:51:59.900 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,300 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> unknown.? -@48904] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:48904/0 2022-03-29T17:51:59.903 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:51:59.904 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.933349609s) 2022-03-29T17:51:59.904 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:51:59.904 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:51:59.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:51:59.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:51:59.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:51:59.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:51:59.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:51:59.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:51:59.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:51:59.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:51:59.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:51:59.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,300 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:51:59.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:51:59.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:51:59.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:51:59.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:51:59.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:51:59.908 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:51:59.909 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,301 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:51:59.909 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:51:59.909 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.909 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.910 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,301 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,301 [shard 0] osd - osd.1: committed_osd_maps(15, 15) 2022-03-29T17:51:59.911 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,301 [shard 0] osd - osd.1: map e 15 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:51:59.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:59.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:59.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:59.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:59.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:59.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:59.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,302 [shard 0] osd - compound_peering_request(id=5, detail=pg_create2(e15 {3.0=14,2022-03-29T17:51:58.272165+0000,3.2=14,2022-03-29T17:51:58.272165+0000,3.4=14,2022-03-29T17:51:58.272165+0000,3.b=14,2022-03-29T17:51:58.272165+0000,3.d=14,2022-03-29T17:51:58.272165+0000,3.10=14,2022-03-29T17:51:58.272165+0000,3.13=14,2022-03-29T17:51:58.272165+0000,3.14=14,2022-03-29T17:51:58.272165+0000,3.19=14,2022-03-29T17:51:58.272165+0000,3.1a=14,2022-03-29T17:51:58.272165+0000,3.1c=14,2022-03-29T17:51:58.272165+0000}) v2): starting 2022-03-29T17:51:59.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - handle_pg_create: 3.0 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=? pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - handle_pg_create: 3.2 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,302 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=? pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.4 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=? pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.b e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=? pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.d e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=? pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.10 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=? pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.13 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=? pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.14 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=? pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:51:59.922 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.831546 2 0.000776 2022-03-29T17:51:59.923 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,303 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4718258, 0.000776664 spent on 2 events 2022-03-29T17:51:59.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:51:59.923 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.934135 0 0.000000 2022-03-29T17:51:59.923 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,303 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3693292, 0.0 spent on 0 events 2022-03-29T17:51:59.924 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:51:59.924 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,303 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:51:59.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:51:59.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:51:59.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:51:59.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.936660767s) 2022-03-29T17:51:59.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:51:59.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:51:59.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.19 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=? pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.1a e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=? pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - handle_pg_create: 3.1c e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:51:59.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,303 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=? pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:51:59.928 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,303 [shard 0] osd - compound_peering_request(id=5, detail=pg_create2(e15 {3.0=14,2022-03-29T17:51:58.272165+0000,3.2=14,2022-03-29T17:51:58.272165+0000,3.4=14,2022-03-29T17:51:58.272165+0000,3.b=14,2022-03-29T17:51:58.272165+0000,3.d=14,2022-03-29T17:51:58.272165+0000,3.10=14,2022-03-29T17:51:58.272165+0000,3.13=14,2022-03-29T17:51:58.272165+0000,3.14=14,2022-03-29T17:51:58.272165+0000,3.19=14,2022-03-29T17:51:58.272165+0000,3.1a=14,2022-03-29T17:51:58.272165+0000,3.1c=14,2022-03-29T17:51:58.272165+0000}) v2): about to fork future 2022-03-29T17:51:59.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:51:59.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:51:59.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:51:59.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:51:59.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:51:59.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:51:59.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:51:59.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:51:59.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:51:59.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:51:59.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:51:59.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:51:59.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:51:59.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_advance_map(id=48, detail=PGAdvanceMap(pg=1.0 from=14 to=15)): start 2022-03-29T17:51:59.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_advance_map(id=49, detail=PGAdvanceMap(pg=2.0 from=14 to=15)): start 2022-03-29T17:51:59.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_advance_map(id=50, detail=PGAdvanceMap(pg=2.1 from=14 to=15)): start 2022-03-29T17:51:59.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,304 [shard 0] osd - pg_advance_map(id=51, detail=PGAdvanceMap(pg=2.2 from=14 to=15)): start 2022-03-29T17:51:59.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=52, detail=PGAdvanceMap(pg=2.3 from=14 to=15)): start 2022-03-29T17:51:59.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=53, detail=PGAdvanceMap(pg=2.4 from=14 to=15)): start 2022-03-29T17:51:59.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=54, detail=PGAdvanceMap(pg=2.5 from=14 to=15)): start 2022-03-29T17:51:59.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=55, detail=PGAdvanceMap(pg=2.6 from=14 to=15)): start 2022-03-29T17:51:59.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=56, detail=PGAdvanceMap(pg=2.7 from=14 to=15)): start 2022-03-29T17:51:59.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=57, detail=PGAdvanceMap(pg=3.0 from=14 to=15)): start 2022-03-29T17:51:59.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=58, detail=PGAdvanceMap(pg=3.2 from=14 to=15)): start 2022-03-29T17:51:59.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=59, detail=PGAdvanceMap(pg=3.4 from=14 to=15)): start 2022-03-29T17:51:59.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=60, detail=PGAdvanceMap(pg=3.b from=14 to=15)): start 2022-03-29T17:51:59.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=61, detail=PGAdvanceMap(pg=3.d from=14 to=15)): start 2022-03-29T17:51:59.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=62, detail=PGAdvanceMap(pg=3.10 from=14 to=15)): start 2022-03-29T17:51:59.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=63, detail=PGAdvanceMap(pg=3.13 from=14 to=15)): start 2022-03-29T17:51:59.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=64, detail=PGAdvanceMap(pg=3.14 from=14 to=15)): start 2022-03-29T17:51:59.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=65, detail=PGAdvanceMap(pg=3.19 from=14 to=15)): start 2022-03-29T17:51:59.941 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.941 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:51:59.941 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:51:59.941 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,305 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:51:59.942 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,305 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:51:59.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:51:59.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:51:59.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=66, detail=PGAdvanceMap(pg=3.1a from=14 to=15)): start 2022-03-29T17:51:59.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,305 [shard 0] osd - pg_advance_map(id=67, detail=PGAdvanceMap(pg=3.1c from=14 to=15)): start 2022-03-29T17:51:59.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:59.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:59.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:59.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:59.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 14 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:59.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:59.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:59.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:59.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 14:11 2022-03-29T17:51:59.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:51:59.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:59.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:59.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:59.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:59.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:59.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:51:59.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:51:59.960 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,306 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:59.960 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.960 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 14:11 2022-03-29T17:51:59.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:51:59.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:51:59.962 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.962 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.962 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.962 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,307 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 14 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:59.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active advmap 2022-03-29T17:51:59.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Started advmap 2022-03-29T17:51:59.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:59.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:51:59.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:59.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:59.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:59.968 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.968 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:51:59.968 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:51:59.969 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.969 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.969 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.969 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.970 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 14 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:59.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:51:59.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:51:59.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,308 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:51:59.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:51:59.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.973 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.973 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:51:59.973 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.973 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.974 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:51:59.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:51:59.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:51:59.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:51:59.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:59.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:59.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 14 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:51:59.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:51:59.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:51:59.979 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.979 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,309 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:51:59.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.917204 2 0.002551 2022-03-29T17:51:59.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.3927863, 0.002551674 spent on 2 events 2022-03-29T17:51:59.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:51:59.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.991865 0 0.000000 2022-03-29T17:51:59.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.318197, 0.0 spent on 0 events 2022-03-29T17:51:59.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:51:59.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:51:59.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:51:59.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:51:59.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:51:59.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.943237305s) 2022-03-29T17:51:59.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:51:59.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:51:59.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:51:59.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 14:11 2022-03-29T17:51:59.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:51:59.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - log is not dirty 2022-03-29T17:51:59.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:51:59.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:51:59.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:51:59.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:51:59.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:51:59.987 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.861229 2 0.002320 2022-03-29T17:51:59.988 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.449453, 0.002320945 spent on 2 events 2022-03-29T17:51:59.988 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:51:59.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:51:59.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:51:59.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:51:59.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:51:59.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:51:59.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:51:59.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:51:59.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:51:59.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:51:59.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:51:59.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:51:59.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:51:59.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:51:59.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:51:59.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:51:59.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:51:59.996 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.854318 2 0.000643 2022-03-29T17:51:59.996 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4570312, 0.000643497 spent on 2 events 2022-03-29T17:51:59.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:51:59.997 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.904409 0 0.000000 2022-03-29T17:51:59.997 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.407067, 0.0 spent on 0 events 2022-03-29T17:51:59.997 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:51:59.997 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:51:59.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:51:59.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:51:59.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:51:59.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.920486450s) 2022-03-29T17:51:59.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:51:59.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:51:59.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,310 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.905576 0 0.000000 2022-03-29T17:52:00.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.405185, 0.0 spent on 0 events 2022-03-29T17:52:00.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,311 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.007873535s) 2022-03-29T17:52:00.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,311 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.006 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.006 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,312 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,312 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.019 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.019 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,313 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.021 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.931416 2 0.003059 2022-03-29T17:52:00.022 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,313 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.3813522, 0.003059981 spent on 2 events 2022-03-29T17:52:00.022 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.022 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.006878 0 0.000000 2022-03-29T17:52:00.022 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,313 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3069973, 0.0 spent on 0 events 2022-03-29T17:52:00.023 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.023 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,313 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.023 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,313 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.023 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.023 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.024 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.947082520s) 2022-03-29T17:52:00.024 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.024 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.024 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.025 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.025 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.025 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.028 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.028 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,314 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 14:11 2022-03-29T17:52:00.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:00.035 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.035 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.036 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.036 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:00.036 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.036 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.037 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.037 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.037 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.037 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.038 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.038 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.038 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.038 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.039 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,314 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.039 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,315 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:00.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:00.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:00.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:00.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.044 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,316 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.044 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map 2022-03-29T17:52:00.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active: handling ActMap 2022-03-29T17:52:00.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: handle ActMap primary 2022-03-29T17:52:00.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean publish_stats_to_osd 14:14 2022-03-29T17:52:00.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.049 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.049 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.049 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.050 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.050 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.050 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.050 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.970030 2 0.002870 2022-03-29T17:52:00.050 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.3471723, 0.002870474 spent on 2 events 2022-03-29T17:52:00.051 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.051 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.052 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,316 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.055 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.863782 2 0.000988 2022-03-29T17:52:00.055 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4539213, 0.000988827 spent on 2 events 2022-03-29T17:52:00.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.057 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.959860 2 0.002096 2022-03-29T17:52:00.058 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.3579333, 0.002096699 spent on 2 events 2022-03-29T17:52:00.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.058 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.021191 0 0.000000 2022-03-29T17:52:00.059 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.2967, 0.0 spent on 0 events 2022-03-29T17:52:00.059 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.059 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,317 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.059 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.059 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.926925659s) 2022-03-29T17:52:00.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.061 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.061 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.061 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.063 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,317 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.064 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.026234 0 0.000000 2022-03-29T17:52:00.064 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,318 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.2920837, 0.0 spent on 0 events 2022-03-29T17:52:00.064 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.064 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,318 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.065 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.065 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.065 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.065 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.951522827s) 2022-03-29T17:52:00.065 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.066 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.066 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.066 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.066 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.067 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.067 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.067 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.067 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.069 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.923869 0 0.000000 2022-03-29T17:52:00.069 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,318 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3939025, 0.0 spent on 0 events 2022-03-29T17:52:00.070 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.070 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,318 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,318 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:52:00.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.014877319s) 2022-03-29T17:52:00.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.073 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.073 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.076 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:00.076 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.076 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.077 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.077 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.077 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.077 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.077 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.078 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.078 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.082 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.082 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,320 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.082 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.082 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.083 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.083 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,319 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.085 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.085 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,320 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.085 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.086 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.086 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.086 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.088 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.088 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.088 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.091 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.091 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,321 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.093 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,320 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:00.093 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.093 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.094 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.094 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.094 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:00.094 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.095 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.095 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.095 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.095 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.096 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.097 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 14 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.098 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:00.098 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:00.098 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.098 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.098 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.099 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.099 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.099 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.899319 2 0.001265 2022-03-29T17:52:00.099 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4228358, 0.001265128 spent on 2 events 2022-03-29T17:52:00.100 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.100 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.972003 0 0.000000 2022-03-29T17:52:00.100 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3502352, 0.0 spent on 0 events 2022-03-29T17:52:00.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:00.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:00.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:00.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 14:11 2022-03-29T17:52:00.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.106 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,321 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.106 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.106 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.916953 2 0.001225 2022-03-29T17:52:00.106 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4059029, 0.001225602 spent on 2 events 2022-03-29T17:52:00.107 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.107 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.008390 0 0.000000 2022-03-29T17:52:00.107 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.314552, 0.0 spent on 0 events 2022-03-29T17:52:00.107 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,322 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.108 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,323 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.108 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.108 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.108 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.108 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.931915283s) 2022-03-29T17:52:00.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.111 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.112 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,323 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.112 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.112 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.112 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.956466675s) 2022-03-29T17:52:00.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.114 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.114 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.114 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.114 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.115 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,323 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 14 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:00.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:00.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:00.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:00.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:00.119 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.119 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.120 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.120 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:00.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.123 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.123 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.123 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:00.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 14:11 2022-03-29T17:52:00.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:00.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.128 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,324 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.128 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.128 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.128 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.130 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.130 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.130 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.131 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.131 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.131 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.131 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,325 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.132 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.132 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.132 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.134 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.134 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.135 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.135 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.135 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.136 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,325 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.136 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.136 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.136 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.136 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.137 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.137 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.137 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.137 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.905267 2 0.002135 2022-03-29T17:52:00.138 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,326 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4209342, 0.00213519 spent on 2 events 2022-03-29T17:52:00.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,325 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.140 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.140 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:00.141 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 14 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:00.143 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.143 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.004348 0 0.000000 2022-03-29T17:52:00.144 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.322643, 0.0 spent on 0 events 2022-03-29T17:52:00.144 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.144 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.144 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.145 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.145 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.145 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.935989380s) 2022-03-29T17:52:00.145 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:00.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:00.152 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - evt epoch is 15, i have 14, will wait 2022-03-29T17:52:00.152 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - osdmap_subscribe(14) 2022-03-29T17:52:00.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.154 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.912378 2 0.001272 2022-03-29T17:52:00.154 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4153028, 0.001272464 spent on 2 events 2022-03-29T17:52:00.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.155 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.020835 0 0.000000 2022-03-29T17:52:00.155 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,327 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3069108, 0.0 spent on 0 events 2022-03-29T17:52:00.157 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,326 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.157 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.157 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.157 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,327 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.158 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.158 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:00.158 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.158 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.159 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.161 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.161 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.162 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.162 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.162 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.165 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.165 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.165 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.165 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.165 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,328 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.166 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.168 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.168 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,328 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,328 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.024856567s) 2022-03-29T17:52:00.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.173 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.173 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.173 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.174 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.174 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.901403 2 0.002635 2022-03-29T17:52:00.174 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,329 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4282334, 0.002635385 spent on 2 events 2022-03-29T17:52:00.174 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.175 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.030934 0 0.000000 2022-03-29T17:52:00.175 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,329 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.298789, 0.0 spent on 0 events 2022-03-29T17:52:00.176 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.177 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.177 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.177 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.177 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.178 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.178 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.178 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 14 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.178 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:00.179 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:00.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.183 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.183 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,330 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.186 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,329 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.186 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,330 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.187 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.187 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.187 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:52:00.188 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.963928223s) 2022-03-29T17:52:00.188 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.188 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.188 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.189 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.189 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.189 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.189 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.190 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.190 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.190 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:00.190 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.191 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.192 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,330 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:00.193 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:00.193 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:00.193 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:00.193 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.194 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 14:11 2022-03-29T17:52:00.194 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.194 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.194 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.195 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.195 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.195 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.199 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.893058 2 0.001136 2022-03-29T17:52:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4390044, 0.001136737 spent on 2 events 2022-03-29T17:52:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.983808 0 0.000000 2022-03-29T17:52:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3483224, 0.0 spent on 0 events 2022-03-29T17:52:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.201 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.203 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.203 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.203 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.205 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.205 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.208 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,331 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.208 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.208 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.894336 2 0.001267 2022-03-29T17:52:00.209 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4385164, 0.001267262 spent on 2 events 2022-03-29T17:52:00.209 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.209 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.029055 0 0.000000 2022-03-29T17:52:00.209 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.303878, 0.0 spent on 0 events 2022-03-29T17:52:00.210 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.210 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,332 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.210 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.210 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.211 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.211 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.941909790s) 2022-03-29T17:52:00.211 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.211 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.211 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.212 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.028182983s) 2022-03-29T17:52:00.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.219 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.219 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.219 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.225 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.226 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,334 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.226 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,333 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.229 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.229 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,334 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,334 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.232 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - pg_advance_map(id=26, detail=PGAdvanceMap(pg=3.8 from=14 to=15)): complete 2022-03-29T17:52:00.232 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.233 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - pg_advance_map(id=22, detail=PGAdvanceMap(pg=2.1 from=14 to=15)): complete 2022-03-29T17:52:00.233 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - pg_advance_map(id=32, detail=PGAdvanceMap(pg=3.1e from=14 to=15)): complete 2022-03-29T17:52:00.233 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.233 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.234 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.235 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.235 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,335 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:00.235 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.235 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.236 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.237 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.969460 2 0.004298 2022-03-29T17:52:00.237 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,336 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.3666854, 0.004298901 spent on 2 events 2022-03-29T17:52:00.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.238 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.036881 0 0.000000 2022-03-29T17:52:00.238 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,336 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.2993374, 0.0 spent on 0 events 2022-03-29T17:52:00.238 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1e 2022-03-29T17:52:00.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_advance_map(id=25, detail=PGAdvanceMap(pg=3.7 from=14 to=15)): complete 2022-03-29T17:52:00.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_advance_map(id=31, detail=PGAdvanceMap(pg=3.1d from=14 to=15)): complete 2022-03-29T17:52:00.242 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.242 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,336 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.244 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.244 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.245 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.245 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.245 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.245 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.246 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.246 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.246 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.246 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,336 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.033203125s) 2022-03-29T17:52:00.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.255 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.256 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.256 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.256 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.889385 2 0.001731 2022-03-29T17:52:00.256 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,338 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4486253, 0.00173118 spent on 2 events 2022-03-29T17:52:00.257 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.257 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.941905 0 0.000000 2022-03-29T17:52:00.257 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,338 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3961685, 0.0 spent on 0 events 2022-03-29T17:52:00.257 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.258 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,338 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.259 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.947067261s) 2022-03-29T17:52:00.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_advance_map(id=21, detail=PGAdvanceMap(pg=2.0 from=14 to=15)): complete 2022-03-29T17:52:00.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1d 2022-03-29T17:52:00.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.8 2022-03-29T17:52:00.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,337 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.264 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.264 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,338 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,338 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.269 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.269 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.269 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.269 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.270 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.270 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.270 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.270 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.272 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.272 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.7 2022-03-29T17:52:00.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_advance_map(id=28, detail=PGAdvanceMap(pg=3.11 from=14 to=15)): complete 2022-03-29T17:52:00.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.18 2022-03-29T17:52:00.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,339 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.279 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.895196 2 0.001888 2022-03-29T17:52:00.280 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,340 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4455345, 0.001888275 spent on 2 events 2022-03-29T17:52:00.281 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.282 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.282 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,340 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.282 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.282 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,340 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:00.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:00.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_advance_map(id=29, detail=PGAdvanceMap(pg=3.16 from=14 to=15)): complete 2022-03-29T17:52:00.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_advance_map(id=30, detail=PGAdvanceMap(pg=3.18 from=14 to=15)): complete 2022-03-29T17:52:00.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.5 2022-03-29T17:52:00.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.290 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.982551 0 0.000000 2022-03-29T17:52:00.290 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,341 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3592925, 0.0 spent on 0 events 2022-03-29T17:52:00.290 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.291 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,341 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,341 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:52:00.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.037933350s) 2022-03-29T17:52:00.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.295 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.296 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.296 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.296 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.296 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:00.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_advance_map(id=27, detail=PGAdvanceMap(pg=3.e from=14 to=15)): complete 2022-03-29T17:52:00.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.16 2022-03-29T17:52:00.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,342 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.11 2022-03-29T17:52:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.307 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.307 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,343 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.308 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,343 [shard 0] osd - pg_advance_map(id=24, detail=PGAdvanceMap(pg=3.5 from=14 to=15)): complete 2022-03-29T17:52:00.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_advance_map(id=23, detail=PGAdvanceMap(pg=2.3 from=14 to=15)): complete 2022-03-29T17:52:00.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.e 2022-03-29T17:52:00.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 2 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.313 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.314 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.314 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.314 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.894518 2 0.001217 2022-03-29T17:52:00.314 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,344 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.449867, 0.001217856 spent on 2 events 2022-03-29T17:52:00.315 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.315 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.990949 0 0.000000 2022-03-29T17:52:00.315 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,344 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3535128, 0.0 spent on 0 events 2022-03-29T17:52:00.315 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.316 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,344 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.316 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.317 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.953430176s) 2022-03-29T17:52:00.317 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.317 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.317 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.317 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - peering_event(id=131, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.320 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.894099 2 0.002267 2022-03-29T17:52:00.322 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,344 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.323 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.323 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.323 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.323 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.324 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.324 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.324 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.324 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.325 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.325 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.325 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.325 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.326 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.326 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.326 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.326 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,345 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.327 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,345 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4512684, 0.002267235 spent on 2 events 2022-03-29T17:52:00.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.328 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,345 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.963366 0 0.000000 2022-03-29T17:52:00.328 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,345 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3824778, 0.0 spent on 0 events 2022-03-29T17:52:00.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,345 [shard 0] osd - peering_event(id=128, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.329 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,345 [shard 0] osd - osd.2: now active 2022-03-29T17:52:00.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - peering_event(id=132, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - peering_event(id=130, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - peering_event(id=134, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - peering_event(id=129, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.333 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.333 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,346 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:52:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.042297363s) 2022-03-29T17:52:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.338 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.338 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,346 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:52:00.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.007873535s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.340 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.340 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,347 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.343 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,347 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.343 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,348 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - peering_event(id=135, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - peering_event(id=136, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - peering_event(id=133, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.007873535s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - Creating 3.19 2022-03-29T17:52:00.348 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.348 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.349 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.349 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,348 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.893203 2 0.001043 2022-03-29T17:52:00.349 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,348 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4557102, 0.001043816 spent on 2 events 2022-03-29T17:52:00.349 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,348 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.350 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.962966 0 0.000000 2022-03-29T17:52:00.350 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,349 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3860211, 0.0 spent on 0 events 2022-03-29T17:52:00.350 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.350 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,349 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.351 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.351 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.351 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.351 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.957992554s) 2022-03-29T17:52:00.352 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.352 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.352 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:00.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:00.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:00.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 14:11 2022-03-29T17:52:00.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 15 2022-03-29T17:52:00.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,349 [shard 0] osd - pg_epoch 14 pg[3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,349 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 14 pg[3.19( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:52:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:52:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.983245850s + 0.000000000s) 2022-03-29T17:52:00.362 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.362 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.362 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.363 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.366 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.366 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,350 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.366 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,350 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.370 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.918152 2 0.002222 2022-03-29T17:52:00.370 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4328892, 0.0022224 spent on 2 events 2022-03-29T17:52:00.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.370 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.023717 0 0.000000 2022-03-29T17:52:00.371 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3273866, 0.0 spent on 0 events 2022-03-29T17:52:00.371 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.371 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_advance_map(id=33, detail=PGAdvanceMap(pg=3.19 from=14 to=15 do_init)): start 2022-03-29T17:52:00.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.001549 0 0.000000 2022-03-29T17:52:00.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - Exiting state: Initial, entered at 1648576319.3499274, 0.0 spent on 0 events 2022-03-29T17:52:00.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.376 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.376 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.376 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.376 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.376 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000131 1 2022-03-29T17:51:59.351637+0000 2022-03-29T17:52:00.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - Exiting state: Reset, entered at 1648576319.3515282, 1648576319.3516378 spent on 1 events 2022-03-29T17:52:00.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.380 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,351 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.383 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.910957 2 0.001203 2022-03-29T17:52:00.383 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,352 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4412723, 0.001203942 spent on 2 events 2022-03-29T17:52:00.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.047134399s) 2022-03-29T17:52:00.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.001062 0 0.000000 2022-03-29T17:52:00.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,352 [shard 0] osd - Exiting state: Start, entered at 1648576319.3517315, 0.0 spent on 0 events 2022-03-29T17:52:00.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,352 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 14 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - Created 3.19 2022-03-29T17:52:00.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,353 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.398 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.010186 0 0.000000 2022-03-29T17:52:00.398 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,353 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3431575, 0.0 spent on 0 events 2022-03-29T17:52:00.398 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.399 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,353 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.399 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.399 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.399 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.400 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.962356567s) 2022-03-29T17:52:00.400 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.400 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.400 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,352 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.404 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.404 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,353 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.007873535s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.007873535s int 16.000000000s) +create_info for pg: 3.19 2022-03-29T17:52:00.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,353 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.413 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.413 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.413 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.413 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,355 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.414 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.414 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.414 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - peering_event(id=340, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,354 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.007873535s int 16.000000000s) 2022-03-29T17:52:00.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 141.988861084s 2022-03-29T17:52:00.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 141.988861084s 2022-03-29T17:52:00.417 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.003052 3 0.001286 2022-03-29T17:52:00.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,355 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.3528602, 0.00128697 spent on 3 events 2022-03-29T17:52:00.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,355 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,355 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.419 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,356 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.007873535s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.420 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.19 2022-03-29T17:52:00.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.425 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_advance_map(id=33, detail=PGAdvanceMap(pg=3.19 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,356 [shard 0] osd - peering_event(id=340, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.427 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.428 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.428 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.924071 2 0.001252 2022-03-29T17:52:00.428 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,357 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.433383, 0.001252517 spent on 2 events 2022-03-29T17:52:00.428 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.428 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.024797 0 0.000000 2022-03-29T17:52:00.429 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,357 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3327289, 0.0 spent on 0 events 2022-03-29T17:52:00.429 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.429 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,357 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.429 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.966506958s) 2022-03-29T17:52:00.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,357 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.915064 2 0.001169 2022-03-29T17:52:00.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,358 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4429476, 0.00116954 spent on 2 events 2022-03-29T17:52:00.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.983303 0 0.000000 2022-03-29T17:52:00.434 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,358 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3747659, 0.0 spent on 0 events 2022-03-29T17:52:00.434 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.434 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,358 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - peering_event(id=137, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.007873535s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.19 2022-03-29T17:52:00.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:00.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.054275513s) 2022-03-29T17:52:00.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.441 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.444 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.444 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.444 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.444 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.444 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.445 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,358 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.448 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.448 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:00.448 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.448 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.449 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.449 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.449 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.449 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.449 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.450 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.450 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.450 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.450 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.451 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,359 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.451 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.452 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.452 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,359 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.452 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,359 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.452 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,360 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.453 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,360 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.453 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,360 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:00.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,360 [shard 0] osd - peering_event(id=138, detail=PeeringEvent(from=2 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.454 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,360 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.454 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,360 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.455 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,360 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.456 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,360 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.456 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.456 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.456 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.457 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.457 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.457 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.457 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.908331 2 0.001190 2022-03-29T17:52:00.457 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,361 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4528713, 0.001190619 spent on 2 events 2022-03-29T17:52:00.458 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Peering advmap 2022-03-29T17:52:00.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering adjust_need_up_thru now 14, need_up_thru now false 2022-03-29T17:52:00.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Started advmap 2022-03-29T17:52:00.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering check_recovery_sources no source osds () went down 2022-03-29T17:52:00.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:52:00.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.024856567s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.462 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,361 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 0.984565 0 0.000000 2022-03-29T17:52:00.462 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,362 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3766997, 0.0 spent on 0 events 2022-03-29T17:52:00.463 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.463 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,362 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.463 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.463 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.463 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.1 ruub 0.000000000s 2022-03-29T17:52:00.464 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.971359253s) 2022-03-29T17:52:00.464 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.464 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.464 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.1 3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.468 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.468 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.468 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,362 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.926952 2 0.001207 2022-03-29T17:52:00.468 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,362 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4360104, 0.001207179 spent on 2 events 2022-03-29T17:52:00.468 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.469 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.025466 0 0.000000 2022-03-29T17:52:00.469 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,363 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3375504, 0.0 spent on 0 events 2022-03-29T17:52:00.469 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.469 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,363 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:52:00.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.059051514s) 2022-03-29T17:52:00.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - peering_event(id=139, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.024856567s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - Creating 3.14 2022-03-29T17:52:00.473 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:00.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,363 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.475 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.476 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.476 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.476 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,364 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.476 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.476 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.477 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.478 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.478 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.482 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.482 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,364 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:52:00.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,364 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.028182983s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.484 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,365 [shard 0] osd - pg_epoch 14 pg[3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.486 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,365 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.920915 2 0.001239 2022-03-29T17:52:00.487 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,365 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.444783, 0.001239625 spent on 2 events 2022-03-29T17:52:00.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,365 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering activate_map 2022-03-29T17:52:00.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: handle ActMap primary 2022-03-29T17:52:00.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering publish_stats_to_osd 14:2 2022-03-29T17:52:00.490 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/WaitUpThru 0.945938 2 0.001812 2022-03-29T17:52:00.490 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576318.4201312, 0.001812712 spent on 2 events 2022-03-29T17:52:00.490 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state: Leaving Peering 2022-03-29T17:52:00.490 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.047616 0 0.000000 2022-03-29T17:52:00.490 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3185124, 0.0 spent on 0 events 2022-03-29T17:52:00.491 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.491 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.491 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.491 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.492 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.0 ruub 0.000000000s 2022-03-29T17:52:00.493 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.028182983s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 14 pg[3.14( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:52:00.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:52:00.495 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering 1.000232 0 0.000000 2022-03-29T17:52:00.495 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576318.3655415, 0.0 spent on 0 events 2022-03-29T17:52:00.495 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active 2022-03-29T17:52:00.496 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,366 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:00.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating state: In Active, about to call activate 2022-03-29T17:52:00.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until peer osd.2 ruub 0.000000000s 2022-03-29T17:52:00.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 141.975814819s) 2022-03-29T17:52:00.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.2 3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 2 2022-03-29T17:52:00.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.501 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.501 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.501 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_advance_map(id=49, detail=PGAdvanceMap(pg=3.17 from=14 to=15)): complete 2022-03-29T17:52:00.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,371 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,372 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,372 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_advance_map(id=35, detail=PGAdvanceMap(pg=2.2 from=14 to=15)): complete 2022-03-29T17:52:00.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_advance_map(id=34, detail=PGAdvanceMap(pg=1.0 from=14 to=15)): complete 2022-03-29T17:52:00.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_advance_map(id=37, detail=PGAdvanceMap(pg=2.5 from=14 to=15)): complete 2022-03-29T17:52:00.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_advance_map(id=38, detail=PGAdvanceMap(pg=2.6 from=14 to=15)): complete 2022-03-29T17:52:00.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_advance_map(id=42, detail=PGAdvanceMap(pg=3.6 from=14 to=15)): complete 2022-03-29T17:52:00.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_advance_map(id=36, detail=PGAdvanceMap(pg=2.4 from=14 to=15)): complete 2022-03-29T17:52:00.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - pg_advance_map(id=50, detail=PGAdvanceMap(pg=3.1b from=14 to=15)): complete 2022-03-29T17:52:00.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - pg_advance_map(id=46, detail=PGAdvanceMap(pg=3.f from=14 to=15)): complete 2022-03-29T17:52:00.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,375 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,375 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.15 2022-03-29T17:52:00.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_advance_map(id=40, detail=PGAdvanceMap(pg=3.1 from=14 to=15)): complete 2022-03-29T17:52:00.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_advance_map(id=48, detail=PGAdvanceMap(pg=3.15 from=14 to=15)): complete 2022-03-29T17:52:00.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_advance_map(id=51, detail=PGAdvanceMap(pg=3.1f from=14 to=15)): complete 2022-03-29T17:52:00.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.6 2022-03-29T17:52:00.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_advance_map(id=39, detail=PGAdvanceMap(pg=2.7 from=14 to=15)): complete 2022-03-29T17:52:00.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1b 2022-03-29T17:52:00.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_advance_map(id=43, detail=PGAdvanceMap(pg=3.9 from=14 to=15)): complete 2022-03-29T17:52:00.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_advance_map(id=45, detail=PGAdvanceMap(pg=3.c from=14 to=15)): complete 2022-03-29T17:52:00.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.17 2022-03-29T17:52:00.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - pg_advance_map(id=44, detail=PGAdvanceMap(pg=3.a from=14 to=15)): complete 2022-03-29T17:52:00.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - pg_advance_map(id=47, detail=PGAdvanceMap(pg=3.12 from=14 to=15)): complete 2022-03-29T17:52:00.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1 2022-03-29T17:52:00.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1f 2022-03-29T17:52:00.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.516 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.516 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.516 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.516 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,382 [shard 0] osd - handle_osd_map osd_map(15..15 src has 1..15) v4 2022-03-29T17:52:00.516 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,382 [shard 0] osd - handle_osd_map epochs [15..15], i have 15, src has [1..15] 2022-03-29T17:52:00.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.f 2022-03-29T17:52:00.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.517 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.518 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.518 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - pg_advance_map(id=41, detail=PGAdvanceMap(pg=3.3 from=14 to=15)): complete 2022-03-29T17:52:00.518 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.c 2022-03-29T17:52:00.518 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.519 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,384 [shard 0] osd - osd.0: now active 2022-03-29T17:52:00.519 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - handle_peering_op on 3.1d from 2 2022-03-29T17:52:00.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931152344s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.a 2022-03-29T17:52:00.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.520 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.9 2022-03-29T17:52:00.521 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.522 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.12 2022-03-29T17:52:00.523 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.523 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.523 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.523 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.3 2022-03-29T17:52:00.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 0 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.524 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931152344s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.525 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - Creating 3.1d 2022-03-29T17:52:00.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - handle_peering_op on 3.1c from 1 2022-03-29T17:52:00.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.014877319s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.526 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.527 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,393 [shard 0] osd - pg_epoch 14 pg[3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.527 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,393 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.527 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - pg_epoch 14 pg[3.1d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 0} acting {2, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 0}+{2, 0} 2022-03-29T17:52:00.528 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000145510} 2022-03-29T17:52:00.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.003128052s + 0.000000000s) 2022-03-29T17:52:00.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.529 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - pg_advance_map(id=52, detail=PGAdvanceMap(pg=3.1d from=14 to=15 do_init)): start 2022-03-29T17:52:00.530 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,395 [shard 0] osd - compound_peering_request(id=4, detail=pg_create2(e15 {3.1=14,2022-03-29T17:51:58.272165+0000,3.3=14,2022-03-29T17:51:58.272165+0000,3.6=14,2022-03-29T17:51:58.272165+0000,3.9=14,2022-03-29T17:51:58.272165+0000,3.a=14,2022-03-29T17:51:58.272165+0000,3.c=14,2022-03-29T17:51:58.272165+0000,3.f=14,2022-03-29T17:51:58.272165+0000,3.12=14,2022-03-29T17:51:58.272165+0000,3.15=14,2022-03-29T17:51:58.272165+0000,3.17=14,2022-03-29T17:51:58.272165+0000,3.1b=14,2022-03-29T17:51:58.272165+0000,3.1f=14,2022-03-29T17:51:58.272165+0000}) v2): starting 2022-03-29T17:52:00.530 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - handle_pg_create: 3.1 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.530 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=? pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.530 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - handle_pg_create: 3.3 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.530 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=? pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - handle_pg_create: 3.6 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=? pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - handle_pg_create: 3.9 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.531 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=? pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.532 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - handle_pg_create: 3.a e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.532 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=? pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.532 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - handle_pg_create: 3.c e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.532 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=? pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating recalc_readable_until readable_until[_ub] 0.000000000s (sent 142.062164307s) 2022-03-29T17:52:00.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate no prior_readable_down_osds to wait on, clearing ub 2022-03-29T17:52:00.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) 2022-03-29T17:52:00.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 is up to date, but sending pg_log anyway 2022-03-29T17:52:00.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 sending log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate peer osd.0 3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) uptodate 2022-03-29T17:52:00.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 0 2022-03-29T17:52:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating activate setting up missing_loc from shard 1 2022-03-29T17:52:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating needs_recovery is recovered 2022-03-29T17:52:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:00.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:3 2022-03-29T17:52:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: Activate Finished 2022-03-29T17:52:00.537 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating 2022-03-29T17:52:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - peering_event(id=341, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - pg_advance_map(id=65, detail=PGAdvanceMap(pg=3.19 from=14 to=15)): complete 2022-03-29T17:52:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,370 [shard 0] osd - peering_event(id=342, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,370 [shard 0] osd - pg_advance_map(id=52, detail=PGAdvanceMap(pg=2.3 from=14 to=15)): complete 2022-03-29T17:52:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,371 [shard 0] osd - peering_event(id=342, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,371 [shard 0] osd - pg_advance_map(id=54, detail=PGAdvanceMap(pg=2.5 from=14 to=15)): complete 2022-03-29T17:52:00.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,371 [shard 0] osd - peering_event(id=343, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,371 [shard 0] osd - peering_event(id=344, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,371 [shard 0] osd - peering_event(id=344, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,372 [shard 0] osd - peering_event(id=341, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,372 [shard 0] osd - pg_advance_map(id=53, detail=PGAdvanceMap(pg=2.4 from=14 to=15)): complete 2022-03-29T17:52:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,372 [shard 0] osd - peering_event(id=343, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,372 [shard 0] osd - pg_advance_map(id=56, detail=PGAdvanceMap(pg=2.7 from=14 to=15)): complete 2022-03-29T17:52:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - peering_event(id=345, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - pg_advance_map(id=64, detail=PGAdvanceMap(pg=3.14 from=14 to=15)): complete 2022-03-29T17:52:00.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - peering_event(id=346, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.542 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,374 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.? -@48904] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:52:00.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - peering_event(id=347, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - pg_advance_map(id=48, detail=PGAdvanceMap(pg=1.0 from=14 to=15)): complete 2022-03-29T17:52:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,375 [shard 0] osd - peering_event(id=345, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,375 [shard 0] osd - peering_event(id=346, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,375 [shard 0] osd - handle_osd_map osd_map(14..15 src has 1..15) v4 2022-03-29T17:52:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,375 [shard 0] osd - handle_osd_map epochs [14..15], i have 15, src has [1..15] 2022-03-29T17:52:00.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,375 [shard 0] osd - peering_event(id=348, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,375 [shard 0] osd - peering_event(id=349, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,375 [shard 0] osd - pg_advance_map(id=60, detail=PGAdvanceMap(pg=3.b from=14 to=15)): complete 2022-03-29T17:52:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.19 2022-03-29T17:52:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - peering_event(id=340, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - peering_event(id=350, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - pg_advance_map(id=67, detail=PGAdvanceMap(pg=3.1c from=14 to=15)): complete 2022-03-29T17:52:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - peering_event(id=348, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - pg_advance_map(id=50, detail=PGAdvanceMap(pg=2.1 from=14 to=15)): complete 2022-03-29T17:52:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - pg_advance_map(id=57, detail=PGAdvanceMap(pg=3.0 from=14 to=15)): complete 2022-03-29T17:52:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - peering_event(id=347, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - peering_event(id=350, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_advance_map(id=66, detail=PGAdvanceMap(pg=3.1a from=14 to=15)): complete 2022-03-29T17:52:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - peering_event(id=349, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_advance_map(id=49, detail=PGAdvanceMap(pg=2.0 from=14 to=15)): complete 2022-03-29T17:52:00.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_advance_map(id=61, detail=PGAdvanceMap(pg=3.d from=14 to=15)): complete 2022-03-29T17:52:00.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_advance_map(id=63, detail=PGAdvanceMap(pg=3.13 from=14 to=15)): complete 2022-03-29T17:52:00.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_advance_map(id=62, detail=PGAdvanceMap(pg=3.10 from=14 to=15)): complete 2022-03-29T17:52:00.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.b 2022-03-29T17:52:00.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - peering_event(id=344, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_advance_map(id=58, detail=PGAdvanceMap(pg=3.2 from=14 to=15)): complete 2022-03-29T17:52:00.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_advance_map(id=51, detail=PGAdvanceMap(pg=2.2 from=14 to=15)): complete 2022-03-29T17:52:00.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:52:00.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.929672241s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1c 2022-03-29T17:52:00.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - peering_event(id=342, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.14 2022-03-29T17:52:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - peering_event(id=341, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - pg_advance_map(id=59, detail=PGAdvanceMap(pg=3.4 from=14 to=15)): complete 2022-03-29T17:52:00.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - pg_advance_map(id=55, detail=PGAdvanceMap(pg=2.6 from=14 to=15)): complete 2022-03-29T17:52:00.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.13 2022-03-29T17:52:00.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - peering_event(id=345, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.2 2022-03-29T17:52:00.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - peering_event(id=346, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.0 2022-03-29T17:52:00.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - peering_event(id=343, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.556 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,384 [shard 0] osd - osd.1: now active 2022-03-29T17:52:00.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.d 2022-03-29T17:52:00.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - peering_event(id=349, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,385 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - peering_event(id=340, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - handle_peering_op on 3.15 from 0 2022-03-29T17:52:00.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.920486450s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=? pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - peering_event(id=342, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.558 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,387 [shard 0] monc - added challenge on [osd.1(client) v2:172.21.15.138:6800/35578 >> client.? -@48904] 2022-03-29T17:52:00.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1a 2022-03-29T17:52:00.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - peering_event(id=347, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.4 2022-03-29T17:52:00.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - peering_event(id=348, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.10 2022-03-29T17:52:00.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: _activate_committed 15 peer_activated now 1 last_interval_started 0 last_epoch_started 0 same_interval_since 14 2022-03-29T17:52:00.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - peering_event(id=350, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=? pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - peering_event(id=344, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=? pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - peering_event(id=345, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - peering_event(id=341, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - peering_event(id=346, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=? pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=? pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - peering_event(id=348, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=? pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - peering_event(id=343, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:52:00.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.933349609s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - peering_event(id=347, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.14 2022-03-29T17:52:00.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=? pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - peering_event(id=349, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=350, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=? pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - handle_authentication client.admin has caps allow * 2022-03-29T17:52:00.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=? pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.4 2022-03-29T17:52:00.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=? pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=? pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=? pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=? pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - handle_peering_op on 3.17 from 0 2022-03-29T17:52:00.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.926925659s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=? pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=? pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.929672241s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=? pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.2 2022-03-29T17:52:00.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=? pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 125.999694824s + 0.000000000s) 2022-03-29T17:52:00.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,366 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_advance_map(id=34, detail=PGAdvanceMap(pg=3.14 from=14 to=15 do_init)): start 2022-03-29T17:52:00.571 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002675 0 0.000000 2022-03-29T17:52:00.571 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,367 [shard 0] osd - Exiting state: Initial, entered at 1648576319.3652196, 0.0 spent on 0 events 2022-03-29T17:52:00.572 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.572 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,367 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,367 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.573 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000128 1 2022-03-29T17:51:59.368064+0000 2022-03-29T17:52:00.573 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - Exiting state: Reset, entered at 1648576319.367958, 1648576319.368064 spent on 1 events 2022-03-29T17:52:00.574 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.574 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.574 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.574 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.574 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000045 0 0.000000 2022-03-29T17:52:00.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,368 [shard 0] osd - Exiting state: Start, entered at 1648576319.3681557, 0.0 spent on 0 events 2022-03-29T17:52:00.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,369 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,369 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - pg_epoch 14 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - Created 3.14 2022-03-29T17:52:00.577 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,369 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,369 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.024856567s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,370 [shard 0] osd - Creating 3.0 2022-03-29T17:52:00.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,370 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,370 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,372 [shard 0] osd - pg_epoch 14 pg[3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,372 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.024856567s int 16.000000000s) +create_info for pg: 3.14 2022-03-29T17:52:00.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.024856567s int 16.000000000s) 2022-03-29T17:52:00.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.006210327s 2022-03-29T17:52:00.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,373 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.006210327s 2022-03-29T17:52:00.581 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,373 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.004985 3 0.001324 2022-03-29T17:52:00.581 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,373 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.3682349, 0.001324856 spent on 3 events 2022-03-29T17:52:00.581 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,374 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.581 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,374 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.582 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,374 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.582 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,374 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.024856567s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:52:00.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,374 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.033203125s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_epoch 14 pg[3.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:52:00.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:52:00.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.009162903s + 0.000000000s) 2022-03-29T17:52:00.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,376 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - pg_advance_map(id=35, detail=PGAdvanceMap(pg=3.0 from=14 to=15 do_init)): start 2022-03-29T17:52:00.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.14 2022-03-29T17:52:00.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.587 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,377 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004708 0 0.000000 2022-03-29T17:52:00.587 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,377 [shard 0] osd - Exiting state: Initial, entered at 1648576319.372946, 0.0 spent on 0 events 2022-03-29T17:52:00.587 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,377 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.587 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,377 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,377 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,378 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.589 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,378 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001176 1 2022-03-29T17:51:59.378860+0000 2022-03-29T17:52:00.589 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,378 [shard 0] osd - Exiting state: Reset, entered at 1648576319.3777072, 1648576319.378861 spent on 1 events 2022-03-29T17:52:00.589 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,378 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.589 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,378 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.589 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,379 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,379 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,379 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,379 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000050 0 0.000000 2022-03-29T17:52:00.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,379 [shard 0] osd - Exiting state: Start, entered at 1648576319.3789902, 0.0 spent on 0 events 2022-03-29T17:52:00.591 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,379 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.591 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,379 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_advance_map(id=34, detail=PGAdvanceMap(pg=3.14 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,379 [shard 0] osd - pg_epoch 14 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - Created 3.0 2022-03-29T17:52:00.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,380 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,380 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.033203125s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - handle_peering_op on 3.9 from 0 2022-03-29T17:52:00.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.941909790s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,381 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.028182983s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,382 [shard 0] osd - Creating 3.1a 2022-03-29T17:52:00.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - peering_event(id=140, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.024856567s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,383 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.14 2022-03-29T17:52:00.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,384 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:00.597 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,385 [shard 0] osd - pg_epoch 14 pg[3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.597 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,385 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - pg_epoch 14 pg[3.1a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:52:00.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:52:00.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.019592285s + 0.000000000s) 2022-03-29T17:52:00.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,386 [shard 0] osd - pg_advance_map(id=36, detail=PGAdvanceMap(pg=3.1a from=14 to=15 do_init)): start 2022-03-29T17:52:00.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - peering_event(id=142, detail=PeeringEvent(from=2 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,387 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.941909790s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:52:00.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,388 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.047134399s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,389 [shard 0] osd - pg_advance_map(id=35, detail=PGAdvanceMap(pg=3.0 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.600 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,389 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004755 0 0.000000 2022-03-29T17:52:00.600 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,389 [shard 0] osd - Exiting state: Initial, entered at 1648576319.3852096, 0.0 spent on 0 events 2022-03-29T17:52:00.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,390 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.602 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000136 1 2022-03-29T17:51:59.390146+0000 2022-03-29T17:52:00.602 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,390 [shard 0] osd - Exiting state: Reset, entered at 1648576319.390031, 1648576319.3901463 spent on 1 events 2022-03-29T17:52:00.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,390 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,390 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,390 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,391 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,391 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000807 0 0.000000 2022-03-29T17:52:00.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,391 [shard 0] osd - Exiting state: Start, entered at 1648576319.3902526, 0.0 spent on 0 events 2022-03-29T17:52:00.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,391 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,391 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - Creating 3.9 2022-03-29T17:52:00.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,391 [shard 0] osd - pg_epoch 14 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.028182983s int 16.000000000s) +create_info for pg: 3.0 2022-03-29T17:52:00.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.028182983s int 16.000000000s) 2022-03-29T17:52:00.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,392 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.025283813s 2022-03-29T17:52:00.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.025283813s 2022-03-29T17:52:00.607 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,393 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.014274 3 0.000283 2022-03-29T17:52:00.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,393 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.3790755, 0.00028367 spent on 3 events 2022-03-29T17:52:00.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,393 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,393 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,393 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,393 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - handle_pg_create: 3.f e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=? pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - handle_pg_create: 3.12 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=? pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - handle_pg_create: 3.15 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=? pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - handle_pg_create: 3.17 e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=? pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - handle_pg_create: 3.1b e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=? pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - handle_pg_create: 3.1f e14 @1648576318.2721653 history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 pi ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=? pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): start 2022-03-29T17:52:00.612 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,397 [shard 0] osd - compound_peering_request(id=4, detail=pg_create2(e15 {3.1=14,2022-03-29T17:51:58.272165+0000,3.3=14,2022-03-29T17:51:58.272165+0000,3.6=14,2022-03-29T17:51:58.272165+0000,3.9=14,2022-03-29T17:51:58.272165+0000,3.a=14,2022-03-29T17:51:58.272165+0000,3.c=14,2022-03-29T17:51:58.272165+0000,3.f=14,2022-03-29T17:51:58.272165+0000,3.12=14,2022-03-29T17:51:58.272165+0000,3.15=14,2022-03-29T17:51:58.272165+0000,3.17=14,2022-03-29T17:51:58.272165+0000,3.1b=14,2022-03-29T17:51:58.272165+0000,3.1f=14,2022-03-29T17:51:58.272165+0000}) v2): about to fork future 2022-03-29T17:52:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004638 0 0.000000 2022-03-29T17:52:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - Exiting state: Initial, entered at 1648576319.393824, 0.0 spent on 0 events 2022-03-29T17:52:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.614 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000125 1 2022-03-29T17:51:59.398626+0000 2022-03-29T17:52:00.615 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - Exiting state: Reset, entered at 1648576319.3985255, 1648576319.398626 spent on 1 events 2022-03-29T17:52:00.615 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.615 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.615 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.616 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.616 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.616 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000042 0 0.000000 2022-03-29T17:52:00.616 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,399 [shard 0] osd - Exiting state: Start, entered at 1648576319.3987293, 0.0 spent on 0 events 2022-03-29T17:52:00.616 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.617 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,399 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:00.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - Created 3.1d 2022-03-29T17:52:00.619 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,400 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.014877319s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - Creating 3.1c 2022-03-29T17:52:00.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931152344s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=? pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - Creating 3.8 2022-03-29T17:52:00.621 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=? pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.028182983s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,393 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,394 [shard 0] osd - peering_event(id=144, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - handle_peering_op on 3.1 from 0 2022-03-29T17:52:00.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947067261s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,395 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - Created 3.1a 2022-03-29T17:52:00.625 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,396 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.0 2022-03-29T17:52:00.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,396 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.033203125s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,397 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.047134399s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,398 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.627 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - pg_epoch 14 pg[3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.627 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,398 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.9( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 2} acting {0, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 2}+{0, 2} 2022-03-29T17:52:00.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000123930} 2022-03-29T17:52:00.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.032577515s + 0.000000000s) 2022-03-29T17:52:00.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - pg_advance_map(id=37, detail=PGAdvanceMap(pg=3.9 from=14 to=15 do_init)): start 2022-03-29T17:52:00.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,399 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,400 [shard 0] osd - Creating 3.2 2022-03-29T17:52:00.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947067261s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.0 2022-03-29T17:52:00.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,401 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:00.631 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,402 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003874 0 0.000000 2022-03-29T17:52:00.632 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,402 [shard 0] osd - Exiting state: Initial, entered at 1648576319.3982038, 0.0 spent on 0 events 2022-03-29T17:52:00.632 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,402 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.632 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,402 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,402 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.633 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,402 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000130 1 2022-03-29T17:51:59.402239+0000 2022-03-29T17:52:00.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,402 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4021392, 1648576319.402239 spent on 1 events 2022-03-29T17:52:00.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,402 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,402 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,403 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,403 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,403 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,403 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000047 0 0.000000 2022-03-29T17:52:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,403 [shard 0] osd - Exiting state: Start, entered at 1648576319.402987, 0.0 spent on 0 events 2022-03-29T17:52:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,403 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,403 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.636 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.636 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.636 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - pg_epoch 14 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:00.636 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.637 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=? pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=? pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.920486450s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=? pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.933349609s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - handle_peering_op on 3.b from 1 2022-03-29T17:52:00.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.037933350s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,403 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=? pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - handle_peering_op on 3.18 from 2 2022-03-29T17:52:00.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.936660767s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - Created 3.9 2022-03-29T17:52:00.643 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,404 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.033203125s int 16.000000000s) +create_info for pg: 3.1a 2022-03-29T17:52:00.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,404 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:52:00.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.943237305s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=? pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.1a 2022-03-29T17:52:00.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.646 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=? pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.647 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 14 pg[3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.647 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,405 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.647 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 14 pg[3.1c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.648 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.648 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:52:00.648 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.648 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:52:00.649 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.014526367s + 0.000000000s) 2022-03-29T17:52:00.649 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.033203125s int 16.000000000s) 2022-03-29T17:52:00.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.038879395s 2022-03-29T17:52:00.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.038879395s 2022-03-29T17:52:00.655 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.014812 3 0.001026 2022-03-29T17:52:00.655 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,405 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.3911047, 0.001026296 spent on 3 events 2022-03-29T17:52:00.656 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.656 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,405 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.656 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,405 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.656 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,406 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - pg_advance_map(id=53, detail=PGAdvanceMap(pg=3.1c from=14 to=15 do_init)): start 2022-03-29T17:52:00.657 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=? pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931152344s int 16.000000000s) +create_info for pg: 3.1d 2022-03-29T17:52:00.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.658 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.931152344s int 16.000000000s) 2022-03-29T17:52:00.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.017196655s 2022-03-29T17:52:00.659 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.017196655s 2022-03-29T17:52:00.660 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.008545 3 0.001292 2022-03-29T17:52:00.660 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,408 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.3998473, 0.001292158 spent on 3 events 2022-03-29T17:52:00.660 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.660 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,408 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.660 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.661 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,408 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.661 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.661 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.661 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.661 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.662 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931152344s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.662 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.662 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004461 0 0.000000 2022-03-29T17:52:00.662 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,408 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4042711, 0.0 spent on 0 events 2022-03-29T17:52:00.663 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,409 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.663 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,409 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.663 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.663 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.663 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.664 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.664 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.665 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,409 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001032 1 2022-03-29T17:51:59.409769+0000 2022-03-29T17:52:00.665 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,409 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4087753, 1648576319.4097693 spent on 1 events 2022-03-29T17:52:00.665 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,409 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.665 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,409 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.666 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,409 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.666 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.666 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.666 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000581 0 0.000000 2022-03-29T17:52:00.666 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - Exiting state: Start, entered at 1648576319.409886, 0.0 spent on 0 events 2022-03-29T17:52:00.667 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.667 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=? pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=? pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - pg_advance_map(id=52, detail=PGAdvanceMap(pg=3.1d from=14 to=15 do_init)): complete 2022-03-29T17:52:00.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - Created 3.1c 2022-03-29T17:52:00.670 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,412 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.c 2022-03-29T17:52:00.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=? pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.014877319s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=? pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=? pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1d 2022-03-29T17:52:00.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931152344s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=? pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.1b 2022-03-29T17:52:00.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=? pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1d 2022-03-29T17:52:00.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:00.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.014877319s int 16.000000000s) +create_info for pg: 3.1c 2022-03-29T17:52:00.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.014877319s int 16.000000000s) 2022-03-29T17:52:00.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.026641846s 2022-03-29T17:52:00.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.026641846s 2022-03-29T17:52:00.678 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.008350 3 0.000831 2022-03-29T17:52:00.679 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,418 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4105191, 0.000831099 spent on 3 events 2022-03-29T17:52:00.679 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.679 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,418 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.679 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.679 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,418 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.014877319s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=? pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,420 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=? pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,420 [shard 0] osd - handle_peering_op on 3.13 from 1 2022-03-29T17:52:00.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,420 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.042297363s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,421 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,421 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=? pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - handle_peering_op on 3.e from 2 2022-03-29T17:52:00.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.963928223s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - pg_advance_map(id=53, detail=PGAdvanceMap(pg=3.1c from=14 to=15 do_init)): complete 2022-03-29T17:52:00.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.1f 2022-03-29T17:52:00.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=? pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=? pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=? pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=? pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.1 2022-03-29T17:52:00.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=? pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.9 2022-03-29T17:52:00.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.686 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=? pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=? pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.014877319s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.687 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=0 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.f 2022-03-29T17:52:00.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=? pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.688 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=? pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=? pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1c 2022-03-29T17:52:00.689 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.690 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=? pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=? pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1c 2022-03-29T17:52:00.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.691 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:00.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=? pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=? pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.692 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=? pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=? pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=0 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=? pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.6 2022-03-29T17:52:00.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=? pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=? pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - handle_peering_op on 3.d from 1 2022-03-29T17:52:00.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.059051514s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=? pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.3 2022-03-29T17:52:00.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=? pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,434 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,434 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=? pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,434 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=? pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,434 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=? pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,434 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.17 2022-03-29T17:52:00.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,434 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=? pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=? pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.936660767s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.037933350s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.a 2022-03-29T17:52:00.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=? pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=? pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.701 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - Creating 3.18 2022-03-29T17:52:00.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.926925659s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - Creating 3.15 2022-03-29T17:52:00.702 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,406 [shard 0] osd - pg_epoch 14 pg[3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.702 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,406 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 14 pg[3.8( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:52:00.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:52:00.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.103981018s + 0.000000000s) 2022-03-29T17:52:00.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_advance_map(id=68, detail=PGAdvanceMap(pg=3.8 from=14 to=15 do_init)): start 2022-03-29T17:52:00.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=? pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): got map 15 2022-03-29T17:52:00.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.943237305s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=? pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=? pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.10 2022-03-29T17:52:00.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=? pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.706 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.707 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.707 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004183 0 0.000000 2022-03-29T17:52:00.707 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4066508, 0.0 spent on 0 events 2022-03-29T17:52:00.707 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.707 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000144 1 2022-03-29T17:51:59.411002+0000 2022-03-29T17:52:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4108868, 1648576319.4110026 spent on 1 events 2022-03-29T17:52:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000043 0 0.000000 2022-03-29T17:52:00.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - Exiting state: Start, entered at 1648576319.4111004, 0.0 spent on 0 events 2022-03-29T17:52:00.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 14 pg[3.15( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:00.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:00.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.108886719s + 0.000000000s) 2022-03-29T17:52:00.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_advance_map(id=69, detail=PGAdvanceMap(pg=3.15 from=14 to=15 do_init)): start 2022-03-29T17:52:00.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_epoch 14 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:00.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - Created 3.8 2022-03-29T17:52:00.715 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,414 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.1c 2022-03-29T17:52:00.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=? pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=? pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.929672241s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=? pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.13 2022-03-29T17:52:00.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=? pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.718 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.005459 0 0.000000 2022-03-29T17:52:00.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,416 [shard 0] osd - Exiting state: Initial, entered at 1648576319.410311, 0.0 spent on 0 events 2022-03-29T17:52:00.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.719 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,416 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000150 1 2022-03-29T17:51:59.416984+0000 2022-03-29T17:52:00.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4168525, 1648576319.4169846 spent on 1 events 2022-03-29T17:52:00.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000043 0 0.000000 2022-03-29T17:52:00.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - Exiting state: Start, entered at 1648576319.417081, 0.0 spent on 0 events 2022-03-29T17:52:00.722 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.723 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=? pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 14 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - Created 3.15 2022-03-29T17:52:00.725 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,418 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - handle_peering_op on 3.6 from 0 2022-03-29T17:52:00.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931915283s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,420 [shard 0] osd - Creating 3.7 2022-03-29T17:52:00.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,420 [shard 0] osd - peering_event(id=357, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,420 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=? pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): pg present 2022-03-29T17:52:00.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,420 [shard 0] osd - peering_event(id=357, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,421 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=? pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,421 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=? pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,421 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.920486450s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - peering_event(id=358, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.19 2022-03-29T17:52:00.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=? pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:52:00.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947082520s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - pg_advance_map(id=68, detail=PGAdvanceMap(pg=3.8 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - Creating 3.17 2022-03-29T17:52:00.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.0 2022-03-29T17:52:00.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=? pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.929672241s int 16.000000000s) +create_info for pg: 3.8 2022-03-29T17:52:00.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.929672241s int 16.000000000s) 2022-03-29T17:52:00.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.121109009s 2022-03-29T17:52:00.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.121109009s 2022-03-29T17:52:00.732 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.014064 3 0.000239 2022-03-29T17:52:00.732 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4111755, 0.000239127 spent on 3 events 2022-03-29T17:52:00.732 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.732 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,426 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.732 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.733 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,426 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.929672241s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931915283s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - peering_event(id=358, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.735 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,428 [shard 0] osd - pg_epoch 14 pg[3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.735 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,428 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.b 2022-03-29T17:52:00.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=? pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - peering_event(id=360, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 14 pg[3.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:52:00.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:52:00.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.125190735s + 0.000000000s) 2022-03-29T17:52:00.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - pg_advance_map(id=70, detail=PGAdvanceMap(pg=3.7 from=14 to=15 do_init)): start 2022-03-29T17:52:00.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - peering_event(id=360, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.738 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=? pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.739 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,430 [shard 0] ms - [osd.1(client) v2:172.21.15.138:6800/35578 >> client.4179 172.21.15.138:0/2702680834@48904] established: gs=0, pgs=3, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:52:00.739 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003617 0 0.000000 2022-03-29T17:52:00.739 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4281244, 0.0 spent on 0 events 2022-03-29T17:52:00.739 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.740 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.741 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000257 1 2022-03-29T17:51:59.432502+0000 2022-03-29T17:52:00.741 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4322665, 1648576319.432502 spent on 1 events 2022-03-29T17:52:00.741 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.742 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:52:00.743 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.743 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.033203125s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.744 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,406 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.744 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 14 pg[3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.745 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,407 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.941909790s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - peering_event(id=141, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.028182983s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 14 pg[3.2( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.745 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:52:00.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,407 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:52:00.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.040817261s + 0.000000000s) 2022-03-29T17:52:00.746 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - pg_advance_map(id=38, detail=PGAdvanceMap(pg=3.2 from=14 to=15 do_init)): start 2022-03-29T17:52:00.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,408 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - Creating 3.1 2022-03-29T17:52:00.747 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - pg_advance_map(id=36, detail=PGAdvanceMap(pg=3.1a from=14 to=15 do_init)): complete 2022-03-29T17:52:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - peering_event(id=146, detail=PeeringEvent(from=2 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,409 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002816 0 0.000000 2022-03-29T17:52:00.748 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4073212, 0.0 spent on 0 events 2022-03-29T17:52:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,410 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.749 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,410 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.750 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.750 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.750 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001172 1 2022-03-29T17:51:59.411329+0000 2022-03-29T17:52:00.750 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - Exiting state: Reset, entered at 1648576319.410185, 1648576319.4113293 spent on 1 events 2022-03-29T17:52:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.751 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.752 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,411 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000045 0 0.000000 2022-03-29T17:52:00.752 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,412 [shard 0] osd - Exiting state: Start, entered at 1648576319.4118419, 0.0 spent on 0 events 2022-03-29T17:52:00.752 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.752 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,412 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.753 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.753 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.753 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,412 [shard 0] osd - handle_peering_op on 3.3 from 0 2022-03-29T17:52:00.753 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.975814819s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.753 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_epoch 14 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1a 2022-03-29T17:52:00.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,413 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:52:00.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,414 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.054275513s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,415 [shard 0] osd - peering_event(id=149, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - peering_event(id=143, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.033203125s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - Created 3.2 2022-03-29T17:52:00.757 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,416 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.941909790s int 16.000000000s) +create_info for pg: 3.9 2022-03-29T17:52:00.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,416 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.941909790s int 16.000000000s) 2022-03-29T17:52:00.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.050842285s 2022-03-29T17:52:00.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.050842285s 2022-03-29T17:52:00.759 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.014797 3 0.001904 2022-03-29T17:52:00.759 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4030762, 0.001904288 spent on 3 events 2022-03-29T17:52:00.760 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.760 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.760 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.760 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,417 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.941909790s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,418 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.047134399s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - pg_advance_map(id=37, detail=PGAdvanceMap(pg=3.9 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1a 2022-03-29T17:52:00.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,419 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:00.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,420 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.764 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,421 [shard 0] osd - pg_epoch 14 pg[3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.764 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,421 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,421 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,421 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.975814819s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,421 [shard 0] osd - peering_event(id=150, detail=PeeringEvent(from=2 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - pg_epoch 14 pg[3.1( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 2} acting {0, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 2}+{0, 2} 2022-03-29T17:52:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000123930} 2022-03-29T17:52:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,422 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.055992126s + 0.000000000s) 2022-03-29T17:52:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - pg_advance_map(id=39, detail=PGAdvanceMap(pg=3.1 from=14 to=15 do_init)): start 2022-03-29T17:52:00.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,423 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.9 2022-03-29T17:52:00.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.768 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,424 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003579 0 0.000000 2022-03-29T17:52:00.768 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,424 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4210021, 0.0 spent on 0 events 2022-03-29T17:52:00.768 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,424 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.769 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,424 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.770 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,424 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.770 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,424 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000116 1 2022-03-29T17:51:59.424728+0000 2022-03-29T17:52:00.770 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,424 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4246309, 1648576319.4247289 spent on 1 events 2022-03-29T17:52:00.770 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.771 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.771 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.771 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.771 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.771 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000054 0 0.000000 2022-03-29T17:52:00.772 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - Exiting state: Start, entered at 1648576319.4258575, 0.0 spent on 0 events 2022-03-29T17:52:00.772 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.772 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,425 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 14 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:00.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,426 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.054275513s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - pg_advance_map(id=38, detail=PGAdvanceMap(pg=3.2 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.047134399s int 16.000000000s) +create_info for pg: 3.2 2022-03-29T17:52:00.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.047134399s int 16.000000000s) 2022-03-29T17:52:00.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.060699463s 2022-03-29T17:52:00.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,427 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.060699463s 2022-03-29T17:52:00.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,427 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.015612 3 0.000833 2022-03-29T17:52:00.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,427 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4120972, 0.000833795 spent on 3 events 2022-03-29T17:52:00.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,427 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,428 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,428 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,428 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,428 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.047134399s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - peering_event(id=145, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.941909790s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - peering_event(id=151, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - Created 3.1 2022-03-29T17:52:00.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,429 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,429 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.9 2022-03-29T17:52:00.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,430 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:00.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947067261s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=? pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - Creating 3.b 2022-03-29T17:52:00.782 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.042297363s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.783 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.15 2022-03-29T17:52:00.783 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.783 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=? pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.783 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.784 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=? pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.784 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - handle_peering_op on 3.10 from 1 2022-03-29T17:52:00.784 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.062164307s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.784 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.963928223s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.784 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.12 2022-03-29T17:52:00.785 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.785 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=? pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.785 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.785 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=? pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.785 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,440 [shard 0] osd - pg_epoch 14 pg[3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.786 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,440 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - pg_epoch 14 pg[3.18( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 0} acting {2, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 0}+{2, 0} 2022-03-29T17:52:00.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000145510} 2022-03-29T17:52:00.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.050758362s + 0.000000000s) 2022-03-29T17:52:00.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - pg_advance_map(id=54, detail=PGAdvanceMap(pg=3.18 from=14 to=15 do_init)): start 2022-03-29T17:52:00.788 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,442 [shard 0] osd - compound_peering_request(id=4, detail=pg_create2(e15 {3.1=14,2022-03-29T17:51:58.272165+0000,3.3=14,2022-03-29T17:51:58.272165+0000,3.6=14,2022-03-29T17:51:58.272165+0000,3.9=14,2022-03-29T17:51:58.272165+0000,3.a=14,2022-03-29T17:51:58.272165+0000,3.c=14,2022-03-29T17:51:58.272165+0000,3.f=14,2022-03-29T17:51:58.272165+0000,3.12=14,2022-03-29T17:51:58.272165+0000,3.15=14,2022-03-29T17:51:58.272165+0000,3.17=14,2022-03-29T17:51:58.272165+0000,3.1b=14,2022-03-29T17:51:58.272165+0000,3.1f=14,2022-03-29T17:51:58.272165+0000}) v2): sub events complete 2022-03-29T17:52:00.788 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,442 [shard 0] osd - pg_epoch 14 pg[3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.788 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,442 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.788 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - compound_peering_request(id=4, detail=pg_create2(e15 {3.1=14,2022-03-29T17:51:58.272165+0000,3.3=14,2022-03-29T17:51:58.272165+0000,3.6=14,2022-03-29T17:51:58.272165+0000,3.9=14,2022-03-29T17:51:58.272165+0000,3.a=14,2022-03-29T17:51:58.272165+0000,3.c=14,2022-03-29T17:51:58.272165+0000,3.f=14,2022-03-29T17:51:58.272165+0000,3.12=14,2022-03-29T17:51:58.272165+0000,3.15=14,2022-03-29T17:51:58.272165+0000,3.17=14,2022-03-29T17:51:58.272165+0000,3.1b=14,2022-03-29T17:51:58.272165+0000,3.1f=14,2022-03-29T17:51:58.272165+0000}) v2): complete 2022-03-29T17:52:00.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:52:00.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:52:00.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.052871704s + 0.000000000s) 2022-03-29T17:52:00.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_advance_map(id=55, detail=PGAdvanceMap(pg=3.b from=14 to=15 do_init)): start 2022-03-29T17:52:00.790 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003527 0 0.000000 2022-03-29T17:52:00.791 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4406302, 0.0 spent on 0 events 2022-03-29T17:52:00.791 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.791 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.792 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000766 1 2022-03-29T17:51:59.444943+0000 2022-03-29T17:52:00.793 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - Exiting state: Reset, entered at 1648576319.444197, 1648576319.4449432 spent on 1 events 2022-03-29T17:52:00.793 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.793 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.793 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.793 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.794 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.794 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000039 0 0.000000 2022-03-29T17:52:00.794 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - Exiting state: Start, entered at 1648576319.4450352, 0.0 spent on 0 events 2022-03-29T17:52:00.794 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.794 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 14 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:00.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - Created 3.18 2022-03-29T17:52:00.797 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,446 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.059051514s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.936660767s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.797 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,446 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004466 0 0.000000 2022-03-29T17:52:00.798 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,446 [shard 0] osd - Exiting state: Initial, entered at 1648576319.442211, 0.0 spent on 0 events 2022-03-29T17:52:00.798 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,446 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.798 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,446 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.798 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.799 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001157 1 2022-03-29T17:51:59.447858+0000 2022-03-29T17:52:00.800 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4467208, 1648576319.4478588 spent on 1 events 2022-03-29T17:52:00.800 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.800 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.800 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.800 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.801 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.801 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,448 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:52:00.801 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,448 [shard 0] osd - Exiting state: Start, entered at 1648576319.4479506, 0.0 spent on 0 events 2022-03-29T17:52:00.801 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,448 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.802 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,448 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - pg_epoch 14 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.803 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.803 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=? pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.803 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.803 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - Created 3.b 2022-03-29T17:52:00.804 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,449 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - Creating 3.e 2022-03-29T17:52:00.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.037933350s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=? pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.805 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,451 [shard 0] osd - pg_epoch 14 pg[3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.805 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,451 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,451 [shard 0] osd - pg_epoch 14 pg[3.e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 0} acting {2, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 0}+{2, 0} 2022-03-29T17:52:00.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000145510} 2022-03-29T17:52:00.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.061019897s + 0.000000000s) 2022-03-29T17:52:00.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_advance_map(id=56, detail=PGAdvanceMap(pg=3.e from=14 to=15 do_init)): start 2022-03-29T17:52:00.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.062164307s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - Creating 3.13 2022-03-29T17:52:00.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - handle_peering_op on 3.9 from 2 2022-03-29T17:52:00.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.941909790s))): start 2022-03-29T17:52:00.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.936660767s int 16.000000000s) +create_info for pg: 3.18 2022-03-29T17:52:00.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.936660767s int 16.000000000s) 2022-03-29T17:52:00.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.063064575s 2022-03-29T17:52:00.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.063064575s 2022-03-29T17:52:00.810 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.009133 3 0.000214 2022-03-29T17:52:00.811 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,454 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4451044, 0.000214715 spent on 3 events 2022-03-29T17:52:00.811 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,455 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.811 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,455 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.811 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,455 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.811 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,455 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.936660767s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.813 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.006222 0 0.000000 2022-03-29T17:52:00.813 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4511235, 0.0 spent on 0 events 2022-03-29T17:52:00.814 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.814 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.815 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000125 1 2022-03-29T17:51:59.457500+0000 2022-03-29T17:52:00.815 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4573977, 1648576319.4575005 spent on 1 events 2022-03-29T17:52:00.815 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.816 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.816 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.816 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.816 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.817 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:52:00.817 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Exiting state: Start, entered at 1648576319.4575822, 0.0 spent on 0 events 2022-03-29T17:52:00.817 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.817 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:00.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_advance_map(id=54, detail=PGAdvanceMap(pg=3.18 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.819 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,459 [shard 0] osd - pg_epoch 14 pg[3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.819 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,459 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.037933350s int 16.000000000s) +create_info for pg: 3.b 2022-03-29T17:52:00.820 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - Exiting state: Start, entered at 1648576319.4325986, 0.0 spent on 0 events 2022-03-29T17:52:00.821 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.821 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,432 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - pg_epoch 14 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:00.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.8 2022-03-29T17:52:00.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - peering_event(id=357, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - Created 3.7 2022-03-29T17:52:00.824 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,434 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,434 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.933349609s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947082520s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=? pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=? pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.920486450s int 16.000000000s) +create_info for pg: 3.15 2022-03-29T17:52:00.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.920486450s int 16.000000000s) 2022-03-29T17:52:00.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.131607056s 2022-03-29T17:52:00.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.131607056s 2022-03-29T17:52:00.827 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.018589 3 0.000228 2022-03-29T17:52:00.827 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,435 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4171546, 0.000228476 spent on 3 events 2022-03-29T17:52:00.827 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,436 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,436 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.920486450s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - pg_advance_map(id=69, detail=PGAdvanceMap(pg=3.15 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 NullEvt +create_info for pg: 3.d 2022-03-29T17:52:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=? pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): submitting ctx transaction 2022-03-29T17:52:00.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - peering_event(id=361, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.831 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,438 [shard 0] osd - pg_epoch 14 pg[3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.831 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,438 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.831 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,438 [shard 0] osd - compound_peering_request(id=5, detail=pg_create2(e15 {3.0=14,2022-03-29T17:51:58.272165+0000,3.2=14,2022-03-29T17:51:58.272165+0000,3.4=14,2022-03-29T17:51:58.272165+0000,3.b=14,2022-03-29T17:51:58.272165+0000,3.d=14,2022-03-29T17:51:58.272165+0000,3.10=14,2022-03-29T17:51:58.272165+0000,3.13=14,2022-03-29T17:51:58.272165+0000,3.14=14,2022-03-29T17:51:58.272165+0000,3.19=14,2022-03-29T17:51:58.272165+0000,3.1a=14,2022-03-29T17:51:58.272165+0000,3.1c=14,2022-03-29T17:51:58.272165+0000}) v2): sub events complete 2022-03-29T17:52:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,438 [shard 0] osd - compound_peering_request(id=5, detail=pg_create2(e15 {3.0=14,2022-03-29T17:51:58.272165+0000,3.2=14,2022-03-29T17:51:58.272165+0000,3.4=14,2022-03-29T17:51:58.272165+0000,3.b=14,2022-03-29T17:51:58.272165+0000,3.d=14,2022-03-29T17:51:58.272165+0000,3.10=14,2022-03-29T17:51:58.272165+0000,3.13=14,2022-03-29T17:51:58.272165+0000,3.14=14,2022-03-29T17:51:58.272165+0000,3.19=14,2022-03-29T17:51:58.272165+0000,3.1a=14,2022-03-29T17:51:58.272165+0000,3.1c=14,2022-03-29T17:51:58.272165+0000}) v2): complete 2022-03-29T17:52:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - peering_event(id=361, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - handle_peering_op on 3.1b from 0 2022-03-29T17:52:00.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.935989380s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - pg_epoch 14 pg[3.17( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:00.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.137268066s + 0.000000000s) 2022-03-29T17:52:00.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_advance_map(id=71, detail=PGAdvanceMap(pg=3.17 from=14 to=15 do_init)): start 2022-03-29T17:52:00.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - peering_event(id=363, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.929672241s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - peering_event(id=363, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=? pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - Creating 3.1e 2022-03-29T17:52:00.835 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004656 0 0.000000 2022-03-29T17:52:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4384646, 0.0 spent on 0 events 2022-03-29T17:52:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.836 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.837 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000118 1 2022-03-29T17:51:59.443555+0000 2022-03-29T17:52:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4434571, 1648576319.443555 spent on 1 events 2022-03-29T17:52:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.838 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.839 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.839 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000039 0 0.000000 2022-03-29T17:52:00.839 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - Exiting state: Start, entered at 1648576319.4436357, 0.0 spent on 0 events 2022-03-29T17:52:00.839 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.839 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.15 2022-03-29T17:52:00.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - peering_event(id=358, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.8 2022-03-29T17:52:00.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - peering_event(id=360, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:00.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_advance_map(id=70, detail=PGAdvanceMap(pg=3.7 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.933349609s int 16.000000000s) +create_info for pg: 3.7 2022-03-29T17:52:00.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.933349609s int 16.000000000s) 2022-03-29T17:52:00.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.142929077s 2022-03-29T17:52:00.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.142929077s 2022-03-29T17:52:00.845 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.014387 3 0.000253 2022-03-29T17:52:00.845 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4326925, 0.000253411 spent on 3 events 2022-03-29T17:52:00.845 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.846 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.846 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.846 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,447 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.933349609s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - Created 3.17 2022-03-29T17:52:00.848 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,448 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=? pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 NullEvt +create_info)): complete 2022-03-29T17:52:00.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.926925659s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - Creating 3.6 2022-03-29T17:52:00.849 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,450 [shard 0] osd - pg_epoch 14 pg[3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.850 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,450 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.935989380s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,451 [shard 0] osd - peering_event(id=364, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,451 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:52:00.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,451 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.951522827s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,451 [shard 0] osd - pg_epoch 14 pg[3.1e( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:52:00.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:52:00.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.148437500s + 0.000000000s) 2022-03-29T17:52:00.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - pg_advance_map(id=72, detail=PGAdvanceMap(pg=3.1e from=14 to=15 do_init)): start 2022-03-29T17:52:00.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - peering_event(id=364, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - peering_event(id=357, detail=PeeringEvent(from=1 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.920486450s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.15 2022-03-29T17:52:00.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - peering_event(id=361, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:00.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - peering_event(id=358, detail=PeeringEvent(from=1 pgid=3.15 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.7 2022-03-29T17:52:00.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - peering_event(id=363, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.856 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,455 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.005106 0 0.000000 2022-03-29T17:52:00.856 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,455 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4501376, 0.0 spent on 0 events 2022-03-29T17:52:00.856 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.857 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.858 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000546 1 2022-03-29T17:51:59.456841+0000 2022-03-29T17:52:00.858 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - Exiting state: Reset, entered at 1648576319.456316, 1648576319.4568415 spent on 1 events 2022-03-29T17:52:00.858 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.859 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.859 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - Creating 3.4 2022-03-29T17:52:00.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,431 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,432 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.860 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,433 [shard 0] osd - Creating 3.3 2022-03-29T17:52:00.861 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,434 [shard 0] osd - pg_epoch 14 pg[3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.861 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,434 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.861 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 14 pg[3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.861 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,435 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - peering_event(id=152, detail=PeeringEvent(from=2 pgid=3.9 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - peering_event(id=147, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.047134399s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 14 pg[3.4( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 2} acting {1, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,435 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 2}+{1, 2} 2022-03-29T17:52:00.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000103e20} 2022-03-29T17:52:00.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.069381714s + 0.000000000s) 2022-03-29T17:52:00.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_advance_map(id=40, detail=PGAdvanceMap(pg=3.4 from=14 to=15 do_init)): start 2022-03-29T17:52:00.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - peering_event(id=155, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.2 2022-03-29T17:52:00.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,436 [shard 0] osd - pg_epoch 14 pg[3.3( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 2} acting {0, 2} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 2}+{0, 2} 2022-03-29T17:52:00.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000123930} 2022-03-29T17:52:00.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.070930481s + 0.000000000s) 2022-03-29T17:52:00.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,437 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - pg_advance_map(id=41, detail=PGAdvanceMap(pg=3.3 from=14 to=15 do_init)): start 2022-03-29T17:52:00.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1 2022-03-29T17:52:00.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,438 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,438 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003230 0 0.000000 2022-03-29T17:52:00.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,438 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4350202, 0.0 spent on 0 events 2022-03-29T17:52:00.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.870 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000160 1 2022-03-29T17:51:59.439467+0000 2022-03-29T17:52:00.870 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - Exiting state: Reset, entered at 1648576319.439325, 1648576319.439467 spent on 1 events 2022-03-29T17:52:00.870 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.870 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000043 0 0.000000 2022-03-29T17:52:00.872 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - Exiting state: Start, entered at 1648576319.439549, 0.0 spent on 0 events 2022-03-29T17:52:00.872 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.872 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,439 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 14 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:00.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,439 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947067261s int 16.000000000s) +create_info for pg: 3.1 2022-03-29T17:52:00.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,440 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.947067261s int 16.000000000s) 2022-03-29T17:52:00.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.074035645s 2022-03-29T17:52:00.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.074035645s 2022-03-29T17:52:00.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.015077 4 2022-03-29T17:51:59.439466+0000 2022-03-29T17:52:00.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,441 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.425955, 1648576319.4394665 spent on 4 events 2022-03-29T17:52:00.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,441 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,441 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,441 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947067261s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.2 2022-03-29T17:52:00.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:00.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,442 [shard 0] osd - pg_advance_map(id=39, detail=PGAdvanceMap(pg=3.1 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.879 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,442 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.008525 0 0.000000 2022-03-29T17:52:00.879 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,442 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4342024, 0.0 spent on 0 events 2022-03-29T17:52:00.879 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.880 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,443 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,443 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.881 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000541 1 2022-03-29T17:51:59.444330+0000 2022-03-29T17:52:00.881 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - Exiting state: Reset, entered at 1648576319.44381, 1648576319.4443307 spent on 1 events 2022-03-29T17:52:00.881 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.882 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.882 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.882 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.882 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.882 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000039 0 0.000000 2022-03-29T17:52:00.883 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - Exiting state: Start, entered at 1648576319.4444306, 0.0 spent on 0 events 2022-03-29T17:52:00.883 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.883 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,444 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 14 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:00.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,444 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - Created 3.4 2022-03-29T17:52:00.885 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,445 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,445 [shard 0] osd - Created 3.3 2022-03-29T17:52:00.887 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,446 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,446 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.975814819s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.054275513s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - peering_event(id=156, detail=PeeringEvent(from=2 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,447 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - handle_peering_op on 3.1d from 0 2022-03-29T17:52:00.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,448 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931152344s))): start 2022-03-29T17:52:00.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,449 [shard 0] osd - peering_event(id=157, detail=PeeringEvent(from=2 pgid=3.1 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - peering_event(id=158, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.054275513s int 16.000000000s) +create_info for pg: 3.4 2022-03-29T17:52:00.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.054275513s int 16.000000000s) 2022-03-29T17:52:00.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.083724976s 2022-03-29T17:52:00.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,450 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.083724976s 2022-03-29T17:52:00.891 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,450 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.011110 3 0.001256 2022-03-29T17:52:00.891 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,450 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4396229, 0.001256163 spent on 3 events 2022-03-29T17:52:00.892 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,451 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.892 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,451 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.892 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,451 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.892 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,451 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,451 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,451 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,451 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.054275513s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,452 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.4 2022-03-29T17:52:00.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931152344s))): got map 15 2022-03-29T17:52:00.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.975814819s int 16.000000000s) +create_info for pg: 3.3 2022-03-29T17:52:00.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.975814819s int 16.000000000s) 2022-03-29T17:52:00.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.086685181s 2022-03-29T17:52:00.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,453 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.086685181s 2022-03-29T17:52:00.897 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,453 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.009193 3 0.000231 2022-03-29T17:52:00.897 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,453 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4444997, 0.000231632 spent on 3 events 2022-03-29T17:52:00.897 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,453 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.898 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,453 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.898 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,453 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.898 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,454 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.037933350s int 16.000000000s) 2022-03-29T17:52:00.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.069305420s 2022-03-29T17:52:00.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.069305420s 2022-03-29T17:52:00.900 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.012455 3 0.000220 2022-03-29T17:52:00.901 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,460 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4480217, 0.000220389 spent on 3 events 2022-03-29T17:52:00.901 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.901 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,460 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.901 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.902 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,460 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.037933350s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - Created 3.e 2022-03-29T17:52:00.904 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,461 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 14 pg[3.13( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:52:00.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:52:00.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.071075439s + 0.000000000s) 2022-03-29T17:52:00.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - pg_advance_map(id=57, detail=PGAdvanceMap(pg=3.13 from=14 to=15 do_init)): start 2022-03-29T17:52:00.906 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004063 0 0.000000 2022-03-29T17:52:00.906 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,463 [shard 0] osd - Exiting state: Initial, entered at 1648576319.459134, 0.0 spent on 0 events 2022-03-29T17:52:00.907 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.907 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,463 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.908 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000771 1 2022-03-29T17:51:59.464008+0000 2022-03-29T17:52:00.908 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4632547, 1648576319.464008 spent on 1 events 2022-03-29T17:52:00.908 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.909 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.909 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.909 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.909 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.910 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000038 0 0.000000 2022-03-29T17:52:00.910 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - Exiting state: Start, entered at 1648576319.4640944, 0.0 spent on 0 events 2022-03-29T17:52:00.910 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.910 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_advance_map(id=55, detail=PGAdvanceMap(pg=3.b from=14 to=15 do_init)): complete 2022-03-29T17:52:00.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.18 2022-03-29T17:52:00.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 14 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:00.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.936660767s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.963928223s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.941909790s))): got map 15 2022-03-29T17:52:00.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - Created 3.13 2022-03-29T17:52:00.915 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,467 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.b 2022-03-29T17:52:00.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.042297363s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.941909790s))): pg present 2022-03-29T17:52:00.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.037933350s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.18 2022-03-29T17:52:00.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:00.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - Creating 3.d 2022-03-29T17:52:00.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.963928223s int 16.000000000s) +create_info for pg: 3.e 2022-03-29T17:52:00.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.963928223s int 16.000000000s) 2022-03-29T17:52:00.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.081008911s 2022-03-29T17:52:00.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.081008911s 2022-03-29T17:52:00.921 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,472 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.014550 3 0.000228 2022-03-29T17:52:00.921 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,472 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4576519, 0.000228052 spent on 3 events 2022-03-29T17:52:00.921 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,472 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.922 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,472 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.922 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,472 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.922 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,472 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,473 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,473 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,473 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,473 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,473 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.963928223s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,473 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=0 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - pg_advance_map(id=56, detail=PGAdvanceMap(pg=3.e from=14 to=15 do_init)): complete 2022-03-29T17:52:00.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.b 2022-03-29T17:52:00.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:00.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - pg_advance_map(id=57, detail=PGAdvanceMap(pg=3.13 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.926 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 14 pg[3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.926 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,477 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 14 pg[3.d( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:52:00.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:52:00.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.086921692s + 0.000000000s) 2022-03-29T17:52:00.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_advance_map(id=58, detail=PGAdvanceMap(pg=3.d from=14 to=15 do_init)): start 2022-03-29T17:52:00.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.e 2022-03-29T17:52:00.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.042297363s int 16.000000000s) +create_info for pg: 3.13 2022-03-29T17:52:00.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.042297363s int 16.000000000s) 2022-03-29T17:52:00.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.088409424s 2022-03-29T17:52:00.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.088409424s 2022-03-29T17:52:00.931 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.015420 3 0.000212 2022-03-29T17:52:00.931 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,479 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4641657, 0.000212581 spent on 3 events 2022-03-29T17:52:00.931 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.932 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,479 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.932 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.932 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,479 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.932 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.932 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.042297363s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.933 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.941909790s) for pg: 3.9 2022-03-29T17:52:00.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 141.941909790s 2022-03-29T17:52:00.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.941909790s (sent 141.941909790s) 2022-03-29T17:52:00.934 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:00.935 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.935 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:00.935 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:00.935 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:00.937 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000040 0 0.000000 2022-03-29T17:52:00.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Exiting state: Start, entered at 1648576319.456951, 0.0 spent on 0 events 2022-03-29T17:52:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,457 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 14 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - Created 3.1e 2022-03-29T17:52:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,458 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,458 [shard 0] osd - pg_epoch 14 pg[3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,458 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.951522827s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_epoch 14 pg[3.6( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:00.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.155487061s + 0.000000000s) 2022-03-29T17:52:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_advance_map(id=73, detail=PGAdvanceMap(pg=3.6 from=14 to=15 do_init)): start 2022-03-29T17:52:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.933349609s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.943237305s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - peering_event(id=366, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - peering_event(id=366, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - handle_peering_op on 3.f from 0 2022-03-29T17:52:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.953430176s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - client_request(id=141, detail=m=[osd_op(client.4179.0:2 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e15) v8]): start 2022-03-29T17:52:00.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - client_request(id=141, detail=m=[osd_op(client.4179.0:2 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e15) v8]): in repeat 2022-03-29T17:52:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - peering_event(id=360, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.926925659s int 16.000000000s) +create_info for pg: 3.17 2022-03-29T17:52:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.926925659s int 16.000000000s) 2022-03-29T17:52:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.159667969s 2022-03-29T17:52:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.159667969s 2022-03-29T17:52:00.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.021175 3 0.000233 2022-03-29T17:52:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4437108, 0.000233549 spent on 3 events 2022-03-29T17:52:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,464 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.926925659s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.006721 0 0.000000 2022-03-29T17:52:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4585397, 0.0 spent on 0 events 2022-03-29T17:52:00.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.951 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000131 1 2022-03-29T17:51:59.466448+0000 2022-03-29T17:52:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4663355, 1648576319.4664485 spent on 1 events 2022-03-29T17:52:00.952 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000054 0 0.000000 2022-03-29T17:52:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - Exiting state: Start, entered at 1648576319.4665282, 0.0 spent on 0 events 2022-03-29T17:52:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,466 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - pg_epoch 14 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:00.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - Created 3.6 2022-03-29T17:52:00.956 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,468 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - peering_event(id=368, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - peering_event(id=369, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - peering_event(id=361, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:00.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - Creating 3.16 2022-03-29T17:52:00.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - peering_event(id=368, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - peering_event(id=369, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931915283s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:00.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - peering_event(id=363, detail=PeeringEvent(from=1 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.7 2022-03-29T17:52:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - peering_event(id=364, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,472 [shard 0] osd - pg_epoch 14 pg[3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,472 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - Creating 3.1b 2022-03-29T17:52:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,473 [shard 0] osd - pg_epoch 14 pg[3.16( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:00.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:00.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:52:00.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:00.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:52:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.170326233s + 0.000000000s) 2022-03-29T17:52:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - pg_advance_map(id=74, detail=PGAdvanceMap(pg=3.16 from=14 to=15 do_init)): start 2022-03-29T17:52:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - client_request(id=141, detail=m=[osd_op(client.4179.0:2 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e15) v8]) same_interval_since: 14 2022-03-29T17:52:00.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - OpSequencer::start_op: op=141, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - peering_event(id=370, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - pg_advance_map(id=71, detail=PGAdvanceMap(pg=3.17 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.943237305s int 16.000000000s) +create_info for pg: 3.1e 2022-03-29T17:52:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:00.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:00.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.943237305s int 16.000000000s) 2022-03-29T17:52:00.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.172042847s 2022-03-29T17:52:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.172042847s 2022-03-29T17:52:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.019155 3 0.000233 2022-03-29T17:52:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,476 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4570212, 0.000233355 spent on 3 events 2022-03-29T17:52:00.965 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,476 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,476 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:00.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.943237305s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.17 2022-03-29T17:52:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - peering_event(id=366, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - peering_event(id=370, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:52:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.956466675s int 16.000000000s) +create_info)): start 2022-03-29T17:52:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - peering_event(id=372, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.953430176s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:00.970 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.007579 0 0.000000 2022-03-29T17:52:00.970 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,480 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4725757, 0.0 spent on 0 events 2022-03-29T17:52:00.970 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:00.972 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:00.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001200 1 2022-03-29T17:51:59.481393+0000 2022-03-29T17:52:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4802322, 1648576319.4813933 spent on 1 events 2022-03-29T17:52:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - Entering state: Started 2022-03-29T17:52:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - Entering state: Start 2022-03-29T17:52:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:52:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - Exiting state: Start, entered at 1648576319.4815352, 0.0 spent on 0 events 2022-03-29T17:52:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,481 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 14 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1e 2022-03-29T17:52:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - peering_event(id=369, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,483 [shard 0] osd - pg_epoch 14 pg[3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,483 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:00.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:00.979 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:00.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:00.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.975814819s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:00.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,454 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - peering_event(id=148, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947067261s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - pg_advance_map(id=41, detail=PGAdvanceMap(pg=3.3 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,455 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:00.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - pg_advance_map(id=40, detail=PGAdvanceMap(pg=3.4 from=14 to=15 do_init)): complete 2022-03-29T17:52:00.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.3 2022-03-29T17:52:00.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:00.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:00.983 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.983 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:00.983 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,456 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931152344s))): pg present 2022-03-29T17:52:00.983 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - peering_event(id=154, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.054275513s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:00.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1 2022-03-29T17:52:00.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,457 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,458 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:00.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.4 2022-03-29T17:52:00.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.986 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.986 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,459 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:00.986 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.3 2022-03-29T17:52:00.986 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:00.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:00.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,460 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:00.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931152344s) for pg: 3.1d 2022-03-29T17:52:00.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,461 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 141.931152344s 2022-03-29T17:52:00.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.931152344s (sent 141.931152344s) 2022-03-29T17:52:00.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:52:00.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:00.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:00.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:00.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 141.931152344s 2022-03-29T17:52:00.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.931152344s (sent 142.095642090s) 2022-03-29T17:52:00.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:00.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:00.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,462 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:00.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:00.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,463 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.096908569s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:00.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - send_pg_created 2022-03-29T17:52:00.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:00.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:00.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:00.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:00.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:00.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:00.996 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:00.996 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:00.996 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:00.996 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:00.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:00.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:00.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:00.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,464 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:00.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:00.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:00.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:00.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:00.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:00.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,465 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:00.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - log is not dirty 2022-03-29T17:52:00.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931152344s))): submitting ctx 2022-03-29T17:52:00.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,466 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:00.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - peering_event(id=159, detail=PeeringEvent(from=2 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - peering_event(id=153, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.975814819s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,467 [shard 0] osd - peering_event(id=162, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - peering_event(id=163, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,468 [shard 0] osd - peering_event(id=160, detail=PeeringEvent(from=2 pgid=3.3 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.001 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - peering_event(id=161, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931152344s))): complete 2022-03-29T17:52:01.001 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.1d 2022-03-29T17:52:01.001 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.170254 3 0.005880 2022-03-29T17:52:01.001 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,469 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.2988517, 0.005880615 spent on 3 events 2022-03-29T17:52:01.002 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.002 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,469 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.002 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.002 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.002 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,469 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.004 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.004 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.004 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,470 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000901 0 0.000000 2022-03-29T17:52:01.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,470 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.4691541, 0.0 spent on 0 events 2022-03-29T17:52:01.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,470 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,470 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,470 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,471 [shard 0] osd - peering_event(id=164, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,472 [shard 0] osd - handle_peering_op on 3.8 from 1 2022-03-29T17:52:01.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,473 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.929672241s))): start 2022-03-29T17:52:01.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - peering_event(id=165, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.929672241s))): got map 15 2022-03-29T17:52:01.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,474 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.929672241s))): pg present 2022-03-29T17:52:01.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - handle_peering_op on 3.18 from 0 2022-03-29T17:52:01.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,475 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.936660767s))): start 2022-03-29T17:52:01.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.936660767s))): got map 15 2022-03-29T17:52:01.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,476 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.929672241s) for pg: 3.8 2022-03-29T17:52:01.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.929672241s 2022-03-29T17:52:01.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.929672241s (sent 141.929672241s) 2022-03-29T17:52:01.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.011 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.011 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.011 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.011 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.012 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.012 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.929672241s 2022-03-29T17:52:01.012 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.929672241s (sent 142.110671997s) 2022-03-29T17:52:01.012 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.012 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,477 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.013 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 141.941909790s 2022-03-29T17:52:01.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.941909790s (sent 142.091354370s) 2022-03-29T17:52:01.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.091606140s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.941909790s))): submitting ctx 2022-03-29T17:52:01.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.023 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.009396 0 0.000000 2022-03-29T17:52:01.023 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4770668, 0.0 spent on 0 events 2022-03-29T17:52:01.023 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.023 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.023 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.025 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000328 1 2022-03-29T17:51:59.486612+0000 2022-03-29T17:52:01.025 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - Exiting state: Reset, entered at 1648576319.4865103, 1648576319.4866126 spent on 1 events 2022-03-29T17:52:01.025 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.025 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.026 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.026 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.026 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.026 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000054 0 0.000000 2022-03-29T17:52:01.026 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - Exiting state: Start, entered at 1648576319.487435, 0.0 spent on 0 events 2022-03-29T17:52:01.027 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.027 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,487 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=0 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.13 2022-03-29T17:52:01.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.029 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 14 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:01.030 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.030 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.030 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.030 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.031 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.031 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - Created 3.d 2022-03-29T17:52:01.031 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,490 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.031 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.031 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.059051514s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.032 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.941909790s))): complete 2022-03-29T17:52:01.032 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.963928223s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.032 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.032 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - Creating 3.10 2022-03-29T17:52:01.033 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.033 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=0 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.033 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.13 2022-03-29T17:52:01.033 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.033 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.034 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.034 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.034 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:01.034 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_advance_map(id=58, detail=PGAdvanceMap(pg=3.d from=14 to=15 do_init)): complete 2022-03-29T17:52:01.035 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.042297363s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.035 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - handle_peering_op on 3.1 from 2 2022-03-29T17:52:01.035 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947067261s))): start 2022-03-29T17:52:01.035 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.e 2022-03-29T17:52:01.035 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.036 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.036 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.036 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.036 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:01.036 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,497 [shard 0] osd - pg_epoch 14 pg[3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:01.037 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,497 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:01.037 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.059051514s int 16.000000000s) +create_info for pg: 3.d 2022-03-29T17:52:01.037 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.037 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.059051514s int 16.000000000s) 2022-03-29T17:52:01.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.107299805s 2022-03-29T17:52:01.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.107299805s 2022-03-29T17:52:01.038 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.010956 3 0.000779 2022-03-29T17:52:01.039 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,498 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4875245, 0.000779642 spent on 3 events 2022-03-29T17:52:01.039 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.039 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,498 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.039 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.040 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,498 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.059051514s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - pg_epoch 14 pg[3.10( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {1, 0} acting {1, 0} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1, 0}+{1, 0} 2022-03-29T17:52:01.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fde50} 2022-03-29T17:52:01.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.109733582s + 0.000000000s) 2022-03-29T17:52:01.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,501 [shard 0] osd - pg_advance_map(id=59, detail=PGAdvanceMap(pg=3.10 from=14 to=15 do_init)): start 2022-03-29T17:52:01.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,501 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.9 2022-03-29T17:52:01.043 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,501 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.166441 4 0.007105 2022-03-29T17:52:01.043 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,501 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3347254, 0.007105202 spent on 4 events 2022-03-29T17:52:01.043 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,501 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.044 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,501 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.044 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,501 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.044 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.044 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.046 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.001300 0 0.000000 2022-03-29T17:52:01.047 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,502 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5012112, 0.0 spent on 0 events 2022-03-29T17:52:01.047 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.047 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,502 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.047 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.047 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.048 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.048 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.048 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.048 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.049 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,503 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.007222 0 0.000000 2022-03-29T17:52:01.049 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,503 [shard 0] osd - Exiting state: Initial, entered at 1648576319.496702, 0.0 spent on 0 events 2022-03-29T17:52:01.049 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.049 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.049 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.050 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.050 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.050 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.050 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.050 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000150 1 2022-03-29T17:51:59.504127+0000 2022-03-29T17:52:01.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - Created 3.16 2022-03-29T17:52:01.052 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,483 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - pg_advance_map(id=73, detail=PGAdvanceMap(pg=3.6 from=14 to=15 do_init)): complete 2022-03-29T17:52:01.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - peering_event(id=364, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_advance_map(id=72, detail=PGAdvanceMap(pg=3.1e from=14 to=15 do_init)): complete 2022-03-29T17:52:01.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - peering_event(id=372, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.17 2022-03-29T17:52:01.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - peering_event(id=368, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:01.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.926925659s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931915283s int 16.000000000s) +create_info for pg: 3.6 2022-03-29T17:52:01.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.931915283s int 16.000000000s) 2022-03-29T17:52:01.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.182083130s 2022-03-29T17:52:01.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.182083130s 2022-03-29T17:52:01.056 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.019605 3 0.000253 2022-03-29T17:52:01.056 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.466612, 0.000253684 spent on 3 events 2022-03-29T17:52:01.057 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.057 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.057 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.057 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,486 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931915283s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.956466675s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:01.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 14 pg[3.1b( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:01.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:01.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.184745789s + 0.000000000s) 2022-03-29T17:52:01.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_advance_map(id=75, detail=PGAdvanceMap(pg=3.1b from=14 to=15 do_init)): start 2022-03-29T17:52:01.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.943237305s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - peering_event(id=373, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.6 2022-03-29T17:52:01.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - peering_event(id=370, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.062 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.007552 0 0.000000 2022-03-29T17:52:01.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,490 [shard 0] osd - Exiting state: Initial, entered at 1648576319.4830875, 0.0 spent on 0 events 2022-03-29T17:52:01.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,490 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.063 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.063 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001159 1 2022-03-29T17:51:59.490775+0000 2022-03-29T17:52:01.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - Exiting state: Reset, entered at 1648576319.490685, 1648576319.4907758 spent on 1 events 2022-03-29T17:52:01.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.066 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.066 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000046 0 0.000000 2022-03-29T17:52:01.066 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,491 [shard 0] osd - Exiting state: Start, entered at 1648576319.4919333, 0.0 spent on 0 events 2022-03-29T17:52:01.066 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,492 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.066 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,492 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - pg_epoch 14 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:01.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - handle_peering_op on 3.a from 0 2022-03-29T17:52:01.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.957992554s int 16.000000000s) +create_info)): start 2022-03-29T17:52:01.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,493 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947082520s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - peering_event(id=373, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - peering_event(id=366, detail=PeeringEvent(from=1 pgid=3.17 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - Creating 3.5 2022-03-29T17:52:01.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - Created 3.1b 2022-03-29T17:52:01.070 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,495 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - peering_event(id=369, detail=PeeringEvent(from=1 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - peering_event(id=375, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.931915283s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - peering_event(id=375, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.935989380s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - peering_event(id=368, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - peering_event(id=376, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.957992554s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:01.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947082520s int 16.000000000s) +create_info for pg: 3.16 2022-03-29T17:52:01.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.947082520s int 16.000000000s) 2022-03-29T17:52:01.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.193695068s 2022-03-29T17:52:01.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.193695068s 2022-03-29T17:52:01.075 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.017254 3 0.000258 2022-03-29T17:52:01.075 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,498 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4816103, 0.00025862 spent on 3 events 2022-03-29T17:52:01.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,499 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,499 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947082520s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - Creating 3.f 2022-03-29T17:52:01.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - peering_event(id=376, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1e 2022-03-29T17:52:01.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - peering_event(id=372, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:01.080 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,501 [shard 0] osd - pg_epoch 14 pg[3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:01.080 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,501 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:01.080 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,501 [shard 0] osd - peering_event(id=370, detail=PeeringEvent(from=1 pgid=3.6 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.080 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 14 pg[3.5( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:52:01.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:52:01.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.198829651s + 0.000000000s) 2022-03-29T17:52:01.082 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.082 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,503 [shard 0] osd - pg_advance_map(id=76, detail=PGAdvanceMap(pg=3.5 from=14 to=15 do_init)): start 2022-03-29T17:52:01.082 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - peering_event(id=377, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.083 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002859 0 0.000000 2022-03-29T17:52:01.083 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Exiting state: Initial, entered at 1648576319.5015385, 0.0 spent on 0 events 2022-03-29T17:52:01.083 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.083 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.086 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000123 1 2022-03-29T17:51:59.504558+0000 2022-03-29T17:52:01.087 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Exiting state: Reset, entered at 1648576319.5044544, 1648576319.5045588 spent on 1 events 2022-03-29T17:52:01.087 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.088 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.088 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.090 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000037 0 0.000000 2022-03-29T17:52:01.090 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Exiting state: Start, entered at 1648576319.5046372, 0.0 spent on 0 events 2022-03-29T17:52:01.090 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.091 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - peering_event(id=377, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - pg_advance_map(id=74, detail=PGAdvanceMap(pg=3.16 from=14 to=15 do_init)): complete 2022-03-29T17:52:01.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 14 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:01.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - Created 3.5 2022-03-29T17:52:01.096 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,507 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.097 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 14 pg[3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:01.098 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,507 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:01.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.951522827s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.6 2022-03-29T17:52:01.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - peering_event(id=373, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:01.102 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.102 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.102 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.103 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.103 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.104 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.112007141s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.104 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,478 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.104 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.105 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.105 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.106 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.106 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.106 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.106 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.107 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.107 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,479 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.107 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.108 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.108 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.108 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.108 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.108 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.109 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.109 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.109 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.109 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.110 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.110 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.110 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.110 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.111 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.111 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.111 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.111 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.929672241s))): submitting ctx 2022-03-29T17:52:01.111 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,480 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.112 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,481 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.112 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,482 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.936660767s))): pg present 2022-03-29T17:52:01.112 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - handle_peering_op on 3.7 from 1 2022-03-29T17:52:01.112 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,483 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.933349609s))): start 2022-03-29T17:52:01.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - peering_event(id=166, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.929672241s))): complete 2022-03-29T17:52:01.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.933349609s))): got map 15 2022-03-29T17:52:01.113 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.8 2022-03-29T17:52:01.113 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.187063 3 0.006199 2022-03-29T17:52:01.114 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,484 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.297505, 0.006199498 spent on 3 events 2022-03-29T17:52:01.114 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.114 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,484 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.114 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.115 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.115 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.115 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.115 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.116 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,484 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.116 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.116 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.116 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.116 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.117 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.117 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000896 0 0.000000 2022-03-29T17:52:01.117 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,485 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.4846175, 0.0 spent on 0 events 2022-03-29T17:52:01.117 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.118 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,485 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.118 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.118 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.118 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.119 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,485 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.119 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.120 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,486 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.120 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.933349609s))): pg present 2022-03-29T17:52:01.120 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.936660767s) for pg: 3.18 2022-03-29T17:52:01.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 141.936660767s 2022-03-29T17:52:01.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.936660767s (sent 141.936660767s) 2022-03-29T17:52:01.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:52:01.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,487 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.123 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.123 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.123 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.123 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.124 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.124 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 141.936660767s 2022-03-29T17:52:01.124 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,488 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.936660767s (sent 142.121215820s) 2022-03-29T17:52:01.124 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.124 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.125 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.125 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.125 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.125 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.122695923s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,489 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.127 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.127 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.127 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.127 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,490 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.129 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.129 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.129 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.129 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.129 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.130 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.130 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.130 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.130 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.131 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.131 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.131 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.131 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,491 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.131 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.132 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.132 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.132 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.936660767s))): submitting ctx 2022-03-29T17:52:01.132 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.133 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,492 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.133 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.933349609s) for pg: 3.7 2022-03-29T17:52:01.133 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.933349609s 2022-03-29T17:52:01.133 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.933349609s (sent 141.933349609s) 2022-03-29T17:52:01.134 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,494 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.134 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.134 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.134 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.134 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.135 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.135 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.135 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.135 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.135 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,495 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.136 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.136 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.136 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.933349609s 2022-03-29T17:52:01.136 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.933349609s (sent 142.129089355s) 2022-03-29T17:52:01.137 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.137 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.137 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.137 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.137 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.138 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.138 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.139 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Exiting state: Reset, entered at 1648576319.5039966, 1648576319.504127 spent on 1 events 2022-03-29T17:52:01.139 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.139 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.140 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.140 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.140 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.140 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,504 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000053 0 0.000000 2022-03-29T17:52:01.141 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,505 [shard 0] osd - Exiting state: Start, entered at 1648576319.5042117, 0.0 spent on 0 events 2022-03-29T17:52:01.141 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,505 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.141 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,505 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.141 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.141 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.142 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=0 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.142 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.142 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.d 2022-03-29T17:52:01.142 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.143 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.143 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.143 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.143 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 14 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:01.143 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.144 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.144 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.144 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947067261s))): got map 15 2022-03-29T17:52:01.144 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.145 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.145 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.145 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - Created 3.10 2022-03-29T17:52:01.145 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,508 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.145 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.059051514s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.147 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - handle_peering_op on 3.15 from 1 2022-03-29T17:52:01.147 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.920486450s))): start 2022-03-29T17:52:01.147 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=0 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.147 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947067261s))): pg present 2022-03-29T17:52:01.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.062164307s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - handle_peering_op on 3.3 from 2 2022-03-29T17:52:01.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.975814819s))): start 2022-03-29T17:52:01.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - pg_advance_map(id=59, detail=PGAdvanceMap(pg=3.10 from=14 to=15 do_init)): complete 2022-03-29T17:52:01.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.d 2022-03-29T17:52:01.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:01.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.10 2022-03-29T17:52:01.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.062164307s int 16.000000000s) +create_info for pg: 3.10 2022-03-29T17:52:01.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.1 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.152 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.152 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 142.062164307s int 16.000000000s) 2022-03-29T17:52:01.152 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.123535156s 2022-03-29T17:52:01.152 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.123535156s 2022-03-29T17:52:01.153 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.009381 4 2022-03-29T17:51:59.514984+0000 2022-03-29T17:52:01.153 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,514 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.5053315, 1648576319.5149841 spent on 4 events 2022-03-29T17:52:01.153 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.153 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,514 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.153 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.154 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,515 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.154 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.154 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.154 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.062164307s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.920486450s))): got map 15 2022-03-29T17:52:01.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947067261s) for pg: 3.1 2022-03-29T17:52:01.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 141.947067261s 2022-03-29T17:52:01.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.947067261s (sent 141.947067261s) 2022-03-29T17:52:01.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:01.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.159 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.159 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 141.947067261s 2022-03-29T17:52:01.159 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.947067261s (sent 142.127700806s) 2022-03-29T17:52:01.159 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.159 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.160 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.160 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.160 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.160 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.160 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.161 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.161 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.127983093s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.161 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.161 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.162 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.162 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.162 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.162 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.162 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.163 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.164 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.165 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.165 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.165 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.165 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.165 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.166 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.166 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.166 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.166 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.167 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.167 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.167 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947067261s))): submitting ctx 2022-03-29T17:52:01.167 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.168 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.168 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,522 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.975814819s))): got map 15 2022-03-29T17:52:01.168 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,522 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.169 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=0 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.169 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.169 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.169 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.920486450s))): pg present 2022-03-29T17:52:01.170 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 1 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 142.062164307s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.170 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.975814819s))): pg present 2022-03-29T17:52:01.170 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.10 2022-03-29T17:52:01.170 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.170 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.171 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.171 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.171 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - dispatch_context_messages sending messages to 1 2022-03-29T17:52:01.171 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - handle_peering_op on 3.1d from 2 2022-03-29T17:52:01.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): start 2022-03-29T17:52:01.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947067261s))): complete 2022-03-29T17:52:01.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.920486450s) for pg: 3.15 2022-03-29T17:52:01.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.920486450s 2022-03-29T17:52:01.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.920486450s (sent 141.920486450s) 2022-03-29T17:52:01.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.174 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.174 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.174 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.174 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - pg_epoch 14 pg[3.f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:01.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:01.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.205848694s + 0.000000000s) 2022-03-29T17:52:01.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - pg_advance_map(id=77, detail=PGAdvanceMap(pg=3.f from=14 to=15 do_init)): start 2022-03-29T17:52:01.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.16 2022-03-29T17:52:01.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - peering_event(id=375, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - pg_advance_map(id=75, detail=PGAdvanceMap(pg=3.1b from=14 to=15 do_init)): complete 2022-03-29T17:52:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,511 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004416 0 0.000000 2022-03-29T17:52:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,511 [shard 0] osd - Exiting state: Initial, entered at 1648576319.5074935, 0.0 spent on 0 events 2022-03-29T17:52:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,511 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.180 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,511 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.183 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000117 1 2022-03-29T17:51:59.512072+0000 2022-03-29T17:52:01.184 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - Exiting state: Reset, entered at 1648576319.511975, 1648576319.5120723 spent on 1 events 2022-03-29T17:52:01.184 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.184 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.184 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.184 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.185 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.185 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000042 0 0.000000 2022-03-29T17:52:01.185 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - Exiting state: Start, entered at 1648576319.5121589, 0.0 spent on 0 events 2022-03-29T17:52:01.185 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.186 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,512 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.935989380s int 16.000000000s) +create_info for pg: 3.1b 2022-03-29T17:52:01.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.935989380s int 16.000000000s) 2022-03-29T17:52:01.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.209014893s 2022-03-29T17:52:01.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.209014893s 2022-03-29T17:52:01.188 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.021125 3 0.000783 2022-03-29T17:52:01.188 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,513 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.4920104, 0.000783544 spent on 3 events 2022-03-29T17:52:01.188 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.188 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,513 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,513 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.935989380s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 14 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:01.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - Created 3.f 2022-03-29T17:52:01.192 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,515 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - Creating 3.11 2022-03-29T17:52:01.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.16 2022-03-29T17:52:01.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - peering_event(id=377, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:01.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - handle_peering_op on 3.19 from 2 2022-03-29T17:52:01.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.007873535s))): start 2022-03-29T17:52:01.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - peering_event(id=379, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.953430176s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1b 2022-03-29T17:52:01.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - peering_event(id=376, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.947082520s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - peering_event(id=379, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - peering_event(id=380, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - peering_event(id=380, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.197 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 14 pg[3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:01.197 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,520 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:01.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 14 pg[3.11( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {2, 1} acting {2, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2, 1}+{2, 1} 2022-03-29T17:52:01.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x607000122e40} 2022-03-29T17:52:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.217208862s + 0.000000000s) 2022-03-29T17:52:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,522 [shard 0] osd - pg_advance_map(id=78, detail=PGAdvanceMap(pg=3.11 from=14 to=15 do_init)): start 2022-03-29T17:52:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,522 [shard 0] osd - Creating 3.a 2022-03-29T17:52:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,522 [shard 0] osd - handle_peering_op on 3.c from 0 2022-03-29T17:52:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.962356567s int 16.000000000s) +create_info)): start 2022-03-29T17:52:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.951522827s int 16.000000000s) +create_info for pg: 3.5 2022-03-29T17:52:01.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.951522827s int 16.000000000s) 2022-03-29T17:52:01.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.220031738s 2022-03-29T17:52:01.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.220031738s 2022-03-29T17:52:01.202 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.019494 3 0.000209 2022-03-29T17:52:01.202 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,524 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.5047035, 0.000209698 spent on 3 events 2022-03-29T17:52:01.202 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.202 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,525 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.203 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.203 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,525 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.951522827s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - peering_event(id=382, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - pg_advance_map(id=76, detail=PGAdvanceMap(pg=3.5 from=14 to=15 do_init)): complete 2022-03-29T17:52:01.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - peering_event(id=383, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - peering_event(id=372, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - peering_event(id=383, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - peering_event(id=382, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,527 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.935989380s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.206 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.008645 0 0.000000 2022-03-29T17:52:01.206 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - Exiting state: Initial, entered at 1648576319.5197515, 0.0 spent on 0 events 2022-03-29T17:52:01.206 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.207 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.208 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000134 1 2022-03-29T17:51:59.528582+0000 2022-03-29T17:52:01.208 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - Exiting state: Reset, entered at 1648576319.5284665, 1648576319.528582 spent on 1 events 2022-03-29T17:52:01.208 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.209 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.209 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.209 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.209 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.209 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000045 0 0.000000 2022-03-29T17:52:01.210 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - Exiting state: Start, entered at 1648576319.5286615, 0.0 spent on 0 events 2022-03-29T17:52:01.210 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.210 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,528 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 14 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:01.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - peering_event(id=373, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - Created 3.11 2022-03-29T17:52:01.212 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,530 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - peering_event(id=375, detail=PeeringEvent(from=1 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.953430176s int 16.000000000s) +create_info for pg: 3.f 2022-03-29T17:52:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.953430176s int 16.000000000s) 2022-03-29T17:52:01.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.227523804s 2022-03-29T17:52:01.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.227523804s 2022-03-29T17:52:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.019433 3 0.000868 2022-03-29T17:52:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,531 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.5122304, 0.000868814 spent on 3 events 2022-03-29T17:52:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,531 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,531 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.953430176s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.007873535s))): got map 15 2022-03-29T17:52:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.956466675s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.218 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,496 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.219 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.130020142s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.220 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.220 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.220 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.220 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,497 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.220 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.221 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.221 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.221 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.221 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.222 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.222 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.223 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.223 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.223 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.224 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.224 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.224 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.224 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.224 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,498 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.226 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.226 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.226 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.226 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.227 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.933349609s))): submitting ctx 2022-03-29T17:52:01.227 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,499 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - peering_event(id=168, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,500 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,501 [shard 0] osd - peering_event(id=167, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.936660767s))): complete 2022-03-29T17:52:01.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,501 [shard 0] osd - peering_event(id=169, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.933349609s))): complete 2022-03-29T17:52:01.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.18 2022-03-29T17:52:01.229 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.196633 3 0.008099 2022-03-29T17:52:01.229 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,502 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.305385, 0.008099379 spent on 3 events 2022-03-29T17:52:01.229 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.229 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,502 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.231 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.231 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.231 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.231 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.232 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,502 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.232 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,503 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.232 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,503 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.232 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,503 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.001320 0 0.000000 2022-03-29T17:52:01.233 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,503 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.502062, 0.0 spent on 0 events 2022-03-29T17:52:01.233 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,503 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.233 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,503 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.234 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,503 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.234 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,503 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.234 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,503 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.234 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.235 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.235 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,504 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.235 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - handle_peering_op on 3.9 from 0 2022-03-29T17:52:01.235 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): start 2022-03-29T17:52:01.235 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,505 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.7 2022-03-29T17:52:01.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.204777 3 0.006590 2022-03-29T17:52:01.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,506 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3015156, 0.006590055 spent on 3 events 2022-03-29T17:52:01.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,506 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.236 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.237 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.237 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.237 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.237 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,506 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.239 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.239 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000673 0 0.000000 2022-03-29T17:52:01.239 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,507 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.506385, 0.0 spent on 0 events 2022-03-29T17:52:01.239 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.240 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,507 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,507 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,508 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): got map 15 2022-03-29T17:52:01.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): pg present 2022-03-29T17:52:01.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - peering_event(id=170, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,509 [shard 0] osd - peering_event(id=171, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - handle_peering_op on 3.1e from 1 2022-03-29T17:52:01.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,510 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.943237305s))): start 2022-03-29T17:52:01.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s) for pg: 3.9 2022-03-29T17:52:01.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s) 2022-03-29T17:52:01.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.144134521s 2022-03-29T17:52:01.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.144134521s 2022-03-29T17:52:01.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,511 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - handle_peering_op on 3.9 from 0 2022-03-29T17:52:01.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): start 2022-03-29T17:52:01.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,512 [shard 0] osd - peering_event(id=172, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): complete 2022-03-29T17:52:01.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.943237305s))): got map 15 2022-03-29T17:52:01.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): got map 15 2022-03-29T17:52:01.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,513 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): pg present 2022-03-29T17:52:01.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.943237305s))): pg present 2022-03-29T17:52:01.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s) for pg: 3.9 2022-03-29T17:52:01.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,514 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.943237305s) for pg: 3.1e 2022-03-29T17:52:01.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.943237305s 2022-03-29T17:52:01.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.943237305s (sent 141.943237305s) 2022-03-29T17:52:01.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,515 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,516 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.943237305s 2022-03-29T17:52:01.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.943237305s (sent 142.150024414s) 2022-03-29T17:52:01.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.150276184s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,517 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.256 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.256 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.256 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.256 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.257 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.920486450s 2022-03-29T17:52:01.257 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.920486450s (sent 142.137985229s) 2022-03-29T17:52:01.257 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.257 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.258 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.259 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.259 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.139762878s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.259 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.259 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.259 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.260 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.260 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.260 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.260 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.261 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.261 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.261 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.261 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.261 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.262 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.262 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.262 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.262 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.263 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.263 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.263 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.263 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.263 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.264 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.264 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.264 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.264 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.264 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.265 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.265 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.920486450s))): submitting ctx 2022-03-29T17:52:01.265 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.265 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.1 2022-03-29T17:52:01.266 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.193017 4 0.006787 2022-03-29T17:52:01.266 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,533 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3408427, 0.006787084 spent on 4 events 2022-03-29T17:52:01.266 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.266 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,533 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.267 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.268 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.269 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.269 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.269 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.269 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000240 0 0.000000 2022-03-29T17:52:01.270 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,534 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5339212, 0.0 spent on 0 events 2022-03-29T17:52:01.270 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.270 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,534 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.270 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.271 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.272 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.272 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.975814819s) for pg: 3.3 2022-03-29T17:52:01.272 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 141.975814819s 2022-03-29T17:52:01.272 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.975814819s (sent 141.975814819s) 2022-03-29T17:52:01.273 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:01.273 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.273 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.273 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.274 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.274 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.274 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.274 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.275 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.275 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.275 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.275 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.275 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 141.975814819s 2022-03-29T17:52:01.276 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.975814819s (sent 142.146469116s) 2022-03-29T17:52:01.276 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.276 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.276 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.277 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.277 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.277 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.277 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.277 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.278 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.147720337s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.278 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.278 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.279 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.279 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.279 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.279 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.280 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.280 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.280 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.280 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.280 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.281 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.281 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.281 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.281 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.282 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.282 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.282 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.283 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.284 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.285 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.975814819s))): submitting ctx 2022-03-29T17:52:01.285 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.285 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.285 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - handle_peering_op on 3.17 from 1 2022-03-29T17:52:01.286 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.926925659s))): start 2022-03-29T17:52:01.286 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.286 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.15 2022-03-29T17:52:01.287 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.229358 4 0.026970 2022-03-29T17:52:01.287 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,544 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3143916, 0.026970381 spent on 4 events 2022-03-29T17:52:01.288 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.288 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,544 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.288 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.288 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.288 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.289 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.290 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.290 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.290 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.290 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.290 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000266 0 0.000000 2022-03-29T17:52:01.291 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,545 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5448298, 0.0 spent on 0 events 2022-03-29T17:52:01.291 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.291 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,545 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.291 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.292 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1b 2022-03-29T17:52:01.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - peering_event(id=379, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:01.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - peering_event(id=384, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.294 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 14 pg[3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:01.294 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,534 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:01.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.5 2022-03-29T17:52:01.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - peering_event(id=380, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - pg_epoch 14 pg[3.a( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:01.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:01.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.232826233s + 0.000000000s) 2022-03-29T17:52:01.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - pg_advance_map(id=79, detail=PGAdvanceMap(pg=3.a from=14 to=15 do_init)): start 2022-03-29T17:52:01.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - peering_event(id=385, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - peering_event(id=384, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.962356567s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:01.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.951522827s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - peering_event(id=385, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - peering_event(id=377, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.007873535s))): pg present 2022-03-29T17:52:01.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - peering_event(id=376, detail=PeeringEvent(from=1 pgid=3.1b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.299 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.003964 0 0.000000 2022-03-29T17:52:01.299 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - Exiting state: Initial, entered at 1648576319.5346694, 0.0 spent on 0 events 2022-03-29T17:52:01.300 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.300 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001152 1 2022-03-29T17:51:59.538776+0000 2022-03-29T17:52:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - Exiting state: Reset, entered at 1648576319.53868, 1648576319.5387762 spent on 1 events 2022-03-29T17:52:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000043 0 0.000000 2022-03-29T17:52:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - Exiting state: Start, entered at 1648576319.5399196, 0.0 spent on 0 events 2022-03-29T17:52:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,540 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 14 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - Created 3.a 2022-03-29T17:52:01.305 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,541 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.956466675s int 16.000000000s) +create_info for pg: 3.11 2022-03-29T17:52:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.2 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.956466675s int 16.000000000s) 2022-03-29T17:52:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.237625122s 2022-03-29T17:52:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.237625122s 2022-03-29T17:52:01.307 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.013028 3 0.000211 2022-03-29T17:52:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,542 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.528736, 0.000211608 spent on 3 events 2022-03-29T17:52:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,542 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.308 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,542 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.956466675s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_advance_map(id=77, detail=PGAdvanceMap(pg=3.f from=14 to=15 do_init)): complete 2022-03-29T17:52:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.5 2022-03-29T17:52:01.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - peering_event(id=382, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:01.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - peering_event(id=379, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.953430176s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - peering_event(id=386, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - peering_event(id=387, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - peering_event(id=386, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.11 2022-03-29T17:52:01.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - peering_event(id=385, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - peering_event(id=387, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - pg_advance_map(id=78, detail=PGAdvanceMap(pg=3.11 from=14 to=15 do_init)): complete 2022-03-29T17:52:01.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.f 2022-03-29T17:52:01.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - peering_event(id=383, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.957992554s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - peering_event(id=380, detail=PeeringEvent(from=1 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.f 2022-03-29T17:52:01.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - peering_event(id=384, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:01.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - handle_peering_op on 3.1f from 0 2022-03-29T17:52:01.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.966506958s int 16.000000000s) +create_info)): start 2022-03-29T17:52:01.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - Creating 3.c 2022-03-29T17:52:01.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.007873535s) for pg: 3.19 2022-03-29T17:52:01.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.007873535s 2022-03-29T17:52:01.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.007873535s (sent 142.007873535s) 2022-03-29T17:52:01.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:01.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.007873535s 2022-03-29T17:52:01.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.007873535s (sent 142.247009277s) 2022-03-29T17:52:01.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.248733521s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - peering_event(id=389, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,518 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,519 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.943237305s))): submitting ctx 2022-03-29T17:52:01.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,520 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - handle_peering_op on 3.e from 0 2022-03-29T17:52:01.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,521 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.963928223s))): start 2022-03-29T17:52:01.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,522 [shard 0] osd - peering_event(id=173, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.943237305s))): complete 2022-03-29T17:52:01.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - peering_event(id=174, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): complete 2022-03-29T17:52:01.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.1e 2022-03-29T17:52:01.339 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,523 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.211214 3 0.006980 2022-03-29T17:52:01.339 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,523 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3123765, 0.00698014 spent on 3 events 2022-03-29T17:52:01.339 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,523 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.340 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,523 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,523 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,524 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.342 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.001483 0 0.000000 2022-03-29T17:52:01.343 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,525 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.523684, 0.0 spent on 0 events 2022-03-29T17:52:01.343 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,525 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.343 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,525 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,526 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.963928223s))): got map 15 2022-03-29T17:52:01.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.963928223s))): pg present 2022-03-29T17:52:01.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,528 [shard 0] osd - peering_event(id=175, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - handle_peering_op on 3.9 from 0 2022-03-29T17:52:01.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): start 2022-03-29T17:52:01.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - handle_peering_op on 3.16 from 1 2022-03-29T17:52:01.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,529 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947082520s))): start 2022-03-29T17:52:01.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.963928223s) for pg: 3.e 2022-03-29T17:52:01.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 141.963928223s 2022-03-29T17:52:01.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.963928223s (sent 141.963928223s) 2022-03-29T17:52:01.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:52:01.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,530 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 141.963928223s 2022-03-29T17:52:01.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,531 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.963928223s (sent 142.164947510s) 2022-03-29T17:52:01.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.351 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.351 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.351 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.351 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.165191650s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,532 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,533 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,534 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.963928223s))): submitting ctx 2022-03-29T17:52:01.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,535 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947082520s))): got map 15 2022-03-29T17:52:01.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,536 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947082520s))): pg present 2022-03-29T17:52:01.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - peering_event(id=176, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.963928223s))): complete 2022-03-29T17:52:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,537 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.e 2022-03-29T17:52:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,537 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.205458 3 0.007450 2022-03-29T17:52:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3325195, 0.00745078 spent on 3 events 2022-03-29T17:52:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.361 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000228 0 0.000000 2022-03-29T17:52:01.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,538 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5380273, 0.0 spent on 0 events 2022-03-29T17:52:01.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,539 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,539 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): got map 15 2022-03-29T17:52:01.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,540 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): pg present 2022-03-29T17:52:01.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947082520s) for pg: 3.16 2022-03-29T17:52:01.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.947082520s 2022-03-29T17:52:01.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.947082520s (sent 141.947082520s) 2022-03-29T17:52:01.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.370 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,546 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): got map 15 2022-03-29T17:52:01.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.371 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): pg present 2022-03-29T17:52:01.372 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.926925659s))): got map 15 2022-03-29T17:52:01.372 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.3 2022-03-29T17:52:01.372 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.179915 4 0.008246 2022-03-29T17:52:01.372 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,548 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3684926, 0.008246825 spent on 4 events 2022-03-29T17:52:01.373 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.373 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,548 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,2 upset 0,2 acting_recovery_backfill 0,2 2022-03-29T17:52:01.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.375 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.376 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.376 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.376 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.376 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000244 0 0.000000 2022-03-29T17:52:01.376 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,548 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5484571, 0.0 spent on 0 events 2022-03-29T17:52:01.377 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.377 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,548 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.377 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.377 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.378 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.378 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.378 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.378 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.378 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - handle_peering_op on 3.1d from 2 2022-03-29T17:52:01.379 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): start 2022-03-29T17:52:01.379 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.975814819s))): complete 2022-03-29T17:52:01.379 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.920486450s))): complete 2022-03-29T17:52:01.380 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.926925659s))): pg present 2022-03-29T17:52:01.380 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s) for pg: 3.1d 2022-03-29T17:52:01.380 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s) 2022-03-29T17:52:01.380 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.161132812s 2022-03-29T17:52:01.380 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.161132812s 2022-03-29T17:52:01.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): got map 15 2022-03-29T17:52:01.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - handle_peering_op on 3.1d from 2 2022-03-29T17:52:01.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): start 2022-03-29T17:52:01.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.926925659s) for pg: 3.17 2022-03-29T17:52:01.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.926925659s 2022-03-29T17:52:01.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.926925659s (sent 141.926925659s) 2022-03-29T17:52:01.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.384 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.384 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.384 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.384 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.384 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.385 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.385 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.385 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.385 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.926925659s 2022-03-29T17:52:01.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.926925659s (sent 142.165069580s) 2022-03-29T17:52:01.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.386 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.387 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.387 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.387 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.388 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.388 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.388 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.166442871s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.388 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.392 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.393 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.393 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.393 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.393 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.394 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.394 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.394 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.394 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.926925659s))): submitting ctx 2022-03-29T17:52:01.395 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.395 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): pg present 2022-03-29T17:52:01.395 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.395 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): got map 15 2022-03-29T17:52:01.395 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.396 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): complete 2022-03-29T17:52:01.396 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): pg present 2022-03-29T17:52:01.396 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.926925659s))): complete 2022-03-29T17:52:01.396 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s) for pg: 3.1d 2022-03-29T17:52:01.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - handle_peering_op on 3.6 from 1 2022-03-29T17:52:01.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931915283s))): start 2022-03-29T17:52:01.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s) for pg: 3.1d 2022-03-29T17:52:01.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.399 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.399 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.399 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.399 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.17 2022-03-29T17:52:01.399 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,566 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.246145 4 0.008588 2022-03-29T17:52:01.400 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,566 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3199933, 0.00858888 spent on 4 events 2022-03-29T17:52:01.400 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,566 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.400 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,566 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.400 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.401 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.402 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.403 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.403 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.001434 0 0.000000 2022-03-29T17:52:01.403 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,567 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5661807, 0.0 spent on 0 events 2022-03-29T17:52:01.403 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.403 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,567 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.404 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.404 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.404 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.404 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.405 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.405 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.405 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931915283s))): got map 15 2022-03-29T17:52:01.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.007873535s))): submitting ctx 2022-03-29T17:52:01.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.11 2022-03-29T17:52:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - peering_event(id=386, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - dispatch_context_messages sending messages to 2 2022-03-29T17:52:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - peering_event(id=389, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 14 pg[3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,557 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - peering_event(id=383, detail=PeeringEvent(from=1 pgid=3.f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.956466675s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_advance_map(id=79, detail=PGAdvanceMap(pg=3.a from=14 to=15 do_init)): complete 2022-03-29T17:52:01.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 14 pg[3.c( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.254463196s + 0.000000000s) 2022-03-29T17:52:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - pg_advance_map(id=80, detail=PGAdvanceMap(pg=3.c from=14 to=15 do_init)): start 2022-03-29T17:52:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.002893 0 0.000000 2022-03-29T17:52:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,559 [shard 0] osd - Exiting state: Initial, entered at 1648576319.5570536, 0.0 spent on 0 events 2022-03-29T17:52:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000154 1 2022-03-29T17:51:59.560424+0000 2022-03-29T17:52:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - Exiting state: Reset, entered at 1648576319.5602925, 1648576319.5604243 spent on 1 events 2022-03-29T17:52:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:52:01.414 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - Exiting state: Start, entered at 1648576319.5605288, 0.0 spent on 0 events 2022-03-29T17:52:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,560 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 14 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - peering_event(id=385, detail=PeeringEvent(from=1 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - Created 3.c 2022-03-29T17:52:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,562 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.a 2022-03-29T17:52:01.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - peering_event(id=387, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.962356567s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - peering_event(id=384, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - peering_event(id=382, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.966506958s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:01.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - peering_event(id=390, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.957992554s int 16.000000000s) +create_info for pg: 3.a 2022-03-29T17:52:01.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.957992554s int 16.000000000s) 2022-03-29T17:52:01.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.261047363s 2022-03-29T17:52:01.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.261047363s 2022-03-29T17:52:01.421 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.025182 4 2022-03-29T17:51:59.564828+0000 2022-03-29T17:52:01.421 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,565 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.5399988, 1648576319.5648286 spent on 4 events 2022-03-29T17:52:01.422 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.422 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,565 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.422 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,565 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.422 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,566 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.957992554s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - peering_event(id=390, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,566 [shard 0] osd - Creating 3.1f 2022-03-29T17:52:01.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - peering_event(id=391, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.007873535s))): complete 2022-03-29T17:52:01.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.19 2022-03-29T17:52:01.425 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.254643 4 0.007584 2022-03-29T17:52:01.425 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,568 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3136957, 0.007584094 spent on 4 events 2022-03-29T17:52:01.425 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.426 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,568 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.428 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000259 0 0.000000 2022-03-29T17:52:01.428 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,568 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5684085, 0.0 spent on 0 events 2022-03-29T17:52:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,568 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - peering_event(id=389, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - peering_event(id=386, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - peering_event(id=391, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - peering_event(id=387, detail=PeeringEvent(from=1 pgid=3.a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.962356567s int 16.000000000s) +create_info for pg: 3.c 2022-03-29T17:52:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.962356567s int 16.000000000s) 2022-03-29T17:52:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.268371582s 2022-03-29T17:52:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.268371582s 2022-03-29T17:52:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.011905 3 0.000254 2022-03-29T17:52:01.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,572 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.5606012, 0.000254755 spent on 3 events 2022-03-29T17:52:01.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,572 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.433 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,572 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.962356567s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.435 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,574 [shard 0] osd - pg_epoch 14 pg[3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:01.436 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,574 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:01.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - pg_advance_map(id=80, detail=PGAdvanceMap(pg=3.c from=14 to=15 do_init)): complete 2022-03-29T17:52:01.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.c 2022-03-29T17:52:01.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - peering_event(id=390, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 14 pg[3.1f( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:01.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:01.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.272209167s + 0.000000000s) 2022-03-29T17:52:01.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - pg_advance_map(id=81, detail=PGAdvanceMap(pg=3.1f from=14 to=15 do_init)): start 2022-03-29T17:52:01.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - peering_event(id=392, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - handle_peering_op on 3.12 from 0 2022-03-29T17:52:01.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.971359253s int 16.000000000s) +create_info)): start 2022-03-29T17:52:01.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,577 [shard 0] osd - peering_event(id=392, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,578 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.957992554s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.441 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,578 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.004202 0 0.000000 2022-03-29T17:52:01.441 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,578 [shard 0] osd - Exiting state: Initial, entered at 1648576319.5739207, 0.0 spent on 0 events 2022-03-29T17:52:01.441 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,578 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.441 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,578 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.441 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.443 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.001206 1 2022-03-29T17:51:59.579388+0000 2022-03-29T17:52:01.443 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - Exiting state: Reset, entered at 1648576319.5782115, 1648576319.5793889 spent on 1 events 2022-03-29T17:52:01.443 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.443 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.444 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.444 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.444 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.444 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:52:01.445 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - Exiting state: Start, entered at 1648576319.579487, 0.0 spent on 0 events 2022-03-29T17:52:01.445 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,541 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.947082520s 2022-03-29T17:52:01.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.947082520s (sent 142.175064087s) 2022-03-29T17:52:01.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,542 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.176330566s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,543 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.457 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,544 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.457 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.457 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.457 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947082520s))): submitting ctx 2022-03-29T17:52:01.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,545 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s) for pg: 3.9 2022-03-29T17:52:01.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - handle_peering_op on 3.1 from 0 2022-03-29T17:52:01.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,547 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): start 2022-03-29T17:52:01.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,548 [shard 0] osd - peering_event(id=179, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.16 2022-03-29T17:52:01.461 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.233888 3 0.007816 2022-03-29T17:52:01.461 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,549 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3156397, 0.007816779 spent on 3 events 2022-03-29T17:52:01.461 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.461 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,549 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.462 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.462 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.462 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,549 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.465 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.001104 0 0.000000 2022-03-29T17:52:01.465 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,550 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.549572, 0.0 spent on 0 events 2022-03-29T17:52:01.465 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.466 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,550 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,550 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - peering_event(id=178, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.947082520s))): complete 2022-03-29T17:52:01.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,551 [shard 0] osd - peering_event(id=177, detail=PeeringEvent(from=0 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.941909790s ub 142.091354370s int 16.000000000s))): complete 2022-03-29T17:52:01.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): got map 15 2022-03-29T17:52:01.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): pg present 2022-03-29T17:52:01.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,552 [shard 0] osd - handle_peering_op on 3.1 from 0 2022-03-29T17:52:01.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,553 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): start 2022-03-29T17:52:01.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - handle_peering_op on 3.5 from 1 2022-03-29T17:52:01.469 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.951522827s))): start 2022-03-29T17:52:01.469 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,554 [shard 0] osd - peering_event(id=180, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.469 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s) for pg: 3.1 2022-03-29T17:52:01.469 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s) 2022-03-29T17:52:01.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.188186646s 2022-03-29T17:52:01.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.188186646s 2022-03-29T17:52:01.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,555 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): got map 15 2022-03-29T17:52:01.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.951522827s))): got map 15 2022-03-29T17:52:01.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.951522827s))): pg present 2022-03-29T17:52:01.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,556 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): pg present 2022-03-29T17:52:01.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s) for pg: 3.1 2022-03-29T17:52:01.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,557 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.473 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - peering_event(id=181, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): complete 2022-03-29T17:52:01.473 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,558 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.951522827s) for pg: 3.5 2022-03-29T17:52:01.473 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.951522827s 2022-03-29T17:52:01.473 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.951522827s (sent 141.951522827s) 2022-03-29T17:52:01.473 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.474 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.474 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.474 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.474 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.475 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.475 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.475 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.476 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.476 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,559 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.476 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.476 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.476 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.951522827s 2022-03-29T17:52:01.477 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.951522827s (sent 142.193099976s) 2022-03-29T17:52:01.477 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.477 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.477 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.478 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.478 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.478 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.478 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,560 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.478 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.479 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.194076538s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.479 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.479 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.479 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,561 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): complete 2022-03-29T17:52:01.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931915283s))): pg present 2022-03-29T17:52:01.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931915283s) for pg: 3.6 2022-03-29T17:52:01.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.931915283s 2022-03-29T17:52:01.483 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.931915283s (sent 141.931915283s) 2022-03-29T17:52:01.484 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.484 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.484 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.484 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.485 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.486 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.931915283s 2022-03-29T17:52:01.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.931915283s (sent 142.180969238s) 2022-03-29T17:52:01.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.487 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.488 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.182312012s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.489 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.490 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.491 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.492 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.492 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.492 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.492 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.493 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.494 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931915283s))): submitting ctx 2022-03-29T17:52:01.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - handle_peering_op on 3.18 from 2 2022-03-29T17:52:01.495 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): start 2022-03-29T17:52:01.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931152344s ub 142.095642090s int 16.000000000s))): complete 2022-03-29T17:52:01.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,577 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,577 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.496 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,578 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): got map 15 2022-03-29T17:52:01.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,578 [shard 0] osd - handle_peering_op on 3.1b from 1 2022-03-29T17:52:01.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.935989380s))): start 2022-03-29T17:52:01.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): pg present 2022-03-29T17:52:01.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.931915283s))): complete 2022-03-29T17:52:01.497 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.6 2022-03-29T17:52:01.498 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,580 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.254344 4 0.008646 2022-03-29T17:52:01.498 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,580 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3259182, 0.008646349 spent on 4 events 2022-03-29T17:52:01.498 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,580 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.498 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,580 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.498 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.499 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.500 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.501 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000699 0 0.000000 2022-03-29T17:52:01.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,581 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5808425, 0.0 spent on 0 events 2022-03-29T17:52:01.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.501 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,581 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.502 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s) for pg: 3.18 2022-03-29T17:52:01.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s) 2022-03-29T17:52:01.503 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.192001343s 2022-03-29T17:52:01.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.192001343s 2022-03-29T17:52:01.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.504 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.935989380s))): got map 15 2022-03-29T17:52:01.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,585 [shard 0] osd - handle_peering_op on 3.18 from 2 2022-03-29T17:52:01.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,585 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): start 2022-03-29T17:52:01.505 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,585 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.935989380s))): pg present 2022-03-29T17:52:01.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,585 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): complete 2022-03-29T17:52:01.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.935989380s) for pg: 3.1b 2022-03-29T17:52:01.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.935989380s 2022-03-29T17:52:01.506 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.935989380s (sent 141.935989380s) 2022-03-29T17:52:01.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.507 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.508 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.509 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.935989380s 2022-03-29T17:52:01.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.935989380s (sent 142.197250366s) 2022-03-29T17:52:01.510 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.511 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.197540283s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.512 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.513 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.514 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.515 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.516 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.516 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.517 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,579 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - peering_event(id=389, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.a 2022-03-29T17:52:01.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - peering_event(id=391, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - pg_epoch 14 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - Created 3.1f 2022-03-29T17:52:01.521 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,582 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.521 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.962356567s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.966506958s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.971359253s int 16.000000000s) +create_info)): got map 15 2022-03-29T17:52:01.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - peering_event(id=394, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - Creating 3.12 2022-03-29T17:52:01.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - peering_event(id=394, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,585 [shard 0] osd - peering_event(id=390, detail=PeeringEvent(from=1 pgid=3.c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.523 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,586 [shard 0] osd - pg_epoch 14 pg[3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial 2022-03-29T17:52:01.524 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,586 [shard 0] osd - Entering state: Initial 2022-03-29T17:52:01.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.c 2022-03-29T17:52:01.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - peering_event(id=392, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.525 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.525 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:01.525 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - peering_event(id=391, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.525 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - handle_peering_op on 3.14 from 2 2022-03-29T17:52:01.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.024856567s))): start 2022-03-29T17:52:01.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.966506958s int 16.000000000s) +create_info for pg: 3.1f 2022-03-29T17:52:01.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.527 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.966506958s int 16.000000000s) 2022-03-29T17:52:01.527 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.284774780s 2022-03-29T17:52:01.527 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.284774780s 2022-03-29T17:52:01.527 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.009340 3 0.000214 2022-03-29T17:52:01.528 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,588 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.579558, 0.000214337 spent on 3 events 2022-03-29T17:52:01.528 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.528 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,588 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.528 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.529 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,588 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.966506958s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 14 pg[3.12( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown init role 1 up {0, 1} acting {0, 1} history ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14 past_intervals ([0,0] all_participants= intervals=) 2022-03-29T17:52:01.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval 2022-03-29T17:52:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval upacting_features 0x3f01cfbb7ffdffff from {0, 1}+{0, 1} 2022-03-29T17:52:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown init_hb_stamps now {0x6070000fc480} 2022-03-29T17:52:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown on_new_interval prior_readable_until_ub 0.000000000s (mnow 126.287055969s + 0.000000000s) 2022-03-29T17:52:01.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_advance_map(id=82, detail=PGAdvanceMap(pg=3.12 from=14 to=15 do_init)): start 2022-03-29T17:52:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_advance_map(id=81, detail=PGAdvanceMap(pg=3.1f from=14 to=15 do_init)): complete 2022-03-29T17:52:01.532 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,592 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,592 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.024856567s))): got map 15 2022-03-29T17:52:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,592 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown exit Initial 0.006592 0 0.000000 2022-03-29T17:52:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,592 [shard 0] osd - Exiting state: Initial, entered at 1648576319.5860064, 0.0 spent on 0 events 2022-03-29T17:52:01.533 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,592 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset 2022-03-29T17:52:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,592 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,592 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown set_last_peering_reset 14 2022-03-29T17:52:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,592 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Clearing blocked outgoing recovery messages 2022-03-29T17:52:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,592 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,592 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown Beginning to block outgoing recovery messages 2022-03-29T17:52:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,592 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,593 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Reset 0.000114 1 2022-03-29T17:51:59.592746+0000 2022-03-29T17:52:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,593 [shard 0] osd - Exiting state: Reset, entered at 1648576319.5926492, 1648576319.592746 spent on 1 events 2022-03-29T17:52:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,593 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:01.535 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,593 [shard 0] osd - Entering state: Started 2022-03-29T17:52:01.536 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,593 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:01.536 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,593 [shard 0] osd - Entering state: Start 2022-03-29T17:52:01.536 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: transitioning to Stray 2022-03-29T17:52:01.536 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Start 0.000458 0 0.000000 2022-03-29T17:52:01.536 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - Exiting state: Start, entered at 1648576319.5938776, 0.0 spent on 0 events 2022-03-29T17:52:01.537 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Stray 2022-03-29T17:52:01.537 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - Entering state: Started/Stray 2022-03-29T17:52:01.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 14 2022-03-29T17:52:01.537 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 14 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:01.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Started advmap 2022-03-29T17:52:01.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:01.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:01.538 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate_map: Not dirtying info: last_persisted is 14 while current is 15 2022-03-29T17:52:01.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - Created 3.12 2022-03-29T17:52:01.539 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - PGAdvanceMap::start new pg pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:01.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.1f 2022-03-29T17:52:01.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - peering_event(id=394, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - peering_event(id=392, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.024856567s))): pg present 2022-03-29T17:52:01.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,596 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.971359253s int 16.000000000s) +create_info)): pg present 2022-03-29T17:52:01.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.966506958s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - peering_event(id=397, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): start 2022-03-29T17:52:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - peering_event(id=397, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): pg present 2022-03-29T17:52:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.024856567s) for pg: 3.14 2022-03-29T17:52:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.024856567s 2022-03-29T17:52:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.024856567s (sent 142.024856567s) 2022-03-29T17:52:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.024856567s 2022-03-29T17:52:01.545 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.024856567s (sent 142.294967651s) 2022-03-29T17:52:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.546 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.547 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.296302795s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.548 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.549 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.550 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - peering_event(id=398, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,562 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.951522827s))): submitting ctx 2022-03-29T17:52:01.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,563 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - handle_peering_op on 3.1 from 0 2022-03-29T17:52:01.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,564 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): start 2022-03-29T17:52:01.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,565 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): got map 15 2022-03-29T17:52:01.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,567 [shard 0] osd - peering_event(id=182, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): complete 2022-03-29T17:52:01.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:52:01.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): start 2022-03-29T17:52:01.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): pg present 2022-03-29T17:52:01.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,568 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.5 2022-03-29T17:52:01.560 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.249528 3 0.008493 2022-03-29T17:52:01.560 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,569 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3197675, 0.008493866 spent on 3 events 2022-03-29T17:52:01.560 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.560 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,569 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.563 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000244 0 0.000000 2022-03-29T17:52:01.563 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,569 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5693645, 0.0 spent on 0 events 2022-03-29T17:52:01.563 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.563 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,569 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,569 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,570 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - peering_event(id=183, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.951522827s))): complete 2022-03-29T17:52:01.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): got map 15 2022-03-29T17:52:01.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,571 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): pg present 2022-03-29T17:52:01.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - handle_peering_op on 3.3 from 0 2022-03-29T17:52:01.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,572 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): start 2022-03-29T17:52:01.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s) for pg: 3.19 2022-03-29T17:52:01.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s) 2022-03-29T17:52:01.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.206268311s 2022-03-29T17:52:01.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,573 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.206268311s 2022-03-29T17:52:01.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:52:01.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,574 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): start 2022-03-29T17:52:01.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s) for pg: 3.1 2022-03-29T17:52:01.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,575 [shard 0] osd - peering_event(id=184, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): got map 15 2022-03-29T17:52:01.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,576 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): got map 15 2022-03-29T17:52:01.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,577 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): pg present 2022-03-29T17:52:01.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,577 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): pg present 2022-03-29T17:52:01.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,578 [shard 0] osd - handle_peering_op on 3.3 from 0 2022-03-29T17:52:01.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,578 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): start 2022-03-29T17:52:01.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,578 [shard 0] osd - peering_event(id=186, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): complete 2022-03-29T17:52:01.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,578 [shard 0] osd - peering_event(id=185, detail=PeeringEvent(from=0 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947067261s ub 142.127700806s int 16.000000000s))): complete 2022-03-29T17:52:01.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,579 [shard 0] osd - handle_peering_op on 3.11 from 1 2022-03-29T17:52:01.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.956466675s))): start 2022-03-29T17:52:01.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): got map 15 2022-03-29T17:52:01.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s) for pg: 3.19 2022-03-29T17:52:01.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,580 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s) for pg: 3.3 2022-03-29T17:52:01.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s) 2022-03-29T17:52:01.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.214263916s 2022-03-29T17:52:01.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,581 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.214263916s 2022-03-29T17:52:01.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,582 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): pg present 2022-03-29T17:52:01.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - peering_event(id=187, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): complete 2022-03-29T17:52:01.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.956466675s))): got map 15 2022-03-29T17:52:01.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - handle_peering_op on 3.3 from 0 2022-03-29T17:52:01.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,583 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): start 2022-03-29T17:52:01.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - peering_event(id=188, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): complete 2022-03-29T17:52:01.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.956466675s))): pg present 2022-03-29T17:52:01.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s) for pg: 3.3 2022-03-29T17:52:01.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,584 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,585 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,585 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): got map 15 2022-03-29T17:52:01.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - peering_event(id=189, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): complete 2022-03-29T17:52:01.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:52:01.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,586 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): start 2022-03-29T17:52:01.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.956466675s) for pg: 3.11 2022-03-29T17:52:01.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,587 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.956466675s 2022-03-29T17:52:01.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.956466675s (sent 141.956466675s) 2022-03-29T17:52:01.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,588 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.956466675s 2022-03-29T17:52:01.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.956466675s (sent 142.221298218s) 2022-03-29T17:52:01.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.222572327s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,589 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.587 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.587 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.588 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.588 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.588 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.588 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.589 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.935989380s))): submitting ctx 2022-03-29T17:52:01.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - handle_peering_op on 3.f from 1 2022-03-29T17:52:01.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.953430176s))): start 2022-03-29T17:52:01.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,592 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,593 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): got map 15 2022-03-29T17:52:01.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,593 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): pg present 2022-03-29T17:52:01.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,593 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.953430176s))): got map 15 2022-03-29T17:52:01.592 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,593 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.1b 2022-03-29T17:52:01.592 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.265063 4 0.005866 2022-03-29T17:52:01.592 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3287106, 0.00586627 spent on 4 events 2022-03-29T17:52:01.593 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.593 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,594 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.593 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.594 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.595 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.595 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000230 0 0.000000 2022-03-29T17:52:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,595 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5948482, 0.0 spent on 0 events 2022-03-29T17:52:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,595 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,596 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,596 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,596 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,596 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.935989380s))): complete 2022-03-29T17:52:01.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,596 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.953430176s))): pg present 2022-03-29T17:52:01.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - handle_peering_op on 3.18 from 2 2022-03-29T17:52:01.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): start 2022-03-29T17:52:01.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s) for pg: 3.18 2022-03-29T17:52:01.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): got map 15 2022-03-29T17:52:01.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.953430176s) for pg: 3.f 2022-03-29T17:52:01.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.953430176s 2022-03-29T17:52:01.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.953430176s (sent 141.953430176s) 2022-03-29T17:52:01.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.602 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.603 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.953430176s 2022-03-29T17:52:01.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.953430176s (sent 142.209915161s) 2022-03-29T17:52:01.604 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.605 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.606 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.606 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.606 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.606 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.211219788s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.607 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.608 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.609 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.610 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.611 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.612 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.613 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.953430176s))): submitting ctx 2022-03-29T17:52:01.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.614 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,605 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): pg present 2022-03-29T17:52:01.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,605 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): complete 2022-03-29T17:52:01.615 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s) for pg: 3.18 2022-03-29T17:52:01.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.616 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.617 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.f 2022-03-29T17:52:01.617 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.261807 4 0.006155 2022-03-29T17:52:01.617 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,607 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3457437, 0.006155014 spent on 4 events 2022-03-29T17:52:01.618 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.618 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,607 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.621 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,608 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.001268 0 0.000000 2022-03-29T17:52:01.621 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,608 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6075954, 0.0 spent on 0 events 2022-03-29T17:52:01.621 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,608 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.621 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,608 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.953430176s))): complete 2022-03-29T17:52:01.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - handle_peering_op on 3.9 from 2 2022-03-29T17:52:01.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.091354370s))): start 2022-03-29T17:52:01.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.024856567s))): submitting ctx 2022-03-29T17:52:01.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.1f 2022-03-29T17:52:01.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:01.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - peering_event(id=394, detail=PeeringEvent(from=1 pgid=3.1f sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - peering_event(id=398, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.971359253s int 16.000000000s) +create_info for pg: 3.12 2022-03-29T17:52:01.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: got info+log from osd.0 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2022-03-29T17:52:01.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - merge_log result log((0'0,0'0], crt=0'0) missing(0 may_include_deletes = 1) changed=0 2022-03-29T17:52:01.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease pg_lease(ru 0.000000000s ub 141.971359253s int 16.000000000s) 2022-03-29T17:52:01.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown no peer_clock_delta_lb -> ruub 142.300567627s 2022-03-29T17:52:01.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown proc_lease readable_until_ub now 142.300567627s 2022-03-29T17:52:01.629 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,604 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown exit Started/Stray 0.010345 3 0.001698 2022-03-29T17:52:01.629 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,604 [shard 0] osd - Exiting state: Started/Stray, entered at 1648576319.5943727, 0.001698484 spent on 3 events 2022-03-29T17:52:01.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,604 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive 2022-03-29T17:52:01.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,605 [shard 0] osd - Entering state: Started/ReplicaActive 2022-03-29T17:52:01.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,605 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.630 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,605 [shard 0] osd - Entering state: Started/ReplicaActive/RepNotRecovering 2022-03-29T17:52:01.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,605 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: In ReplicaActive, about to call activate 2022-03-29T17:52:01.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,605 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:01.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,605 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Activate Finished 2022-03-29T17:52:01.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.971359253s int 16.000000000s) +create_info)): submitting ctx 2022-03-29T17:52:01.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - peering_event(id=399, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): start 2022-03-29T17:52:01.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,607 [shard 0] osd - peering_event(id=399, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): pg present 2022-03-29T17:52:01.632 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - pg_advance_map(id=82, detail=PGAdvanceMap(pg=3.12 from=14 to=15 do_init)): complete 2022-03-29T17:52:01.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.024856567s))): complete 2022-03-29T17:52:01.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.14 2022-03-29T17:52:01.633 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,608 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.277648 4 0.007338 2022-03-29T17:52:01.634 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,608 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3305304, 0.007338619 spent on 4 events 2022-03-29T17:52:01.634 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,608 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.634 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,608 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.637 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.001288 0 0.000000 2022-03-29T17:52:01.637 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,609 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6082225, 0.0 spent on 0 events 2022-03-29T17:52:01.638 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.638 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,609 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - peering_event(id=398, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 IntervalFlush for pg: 3.12 2022-03-29T17:52:01.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: Ending blocked outgoing recovery messages 2022-03-29T17:52:01.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,611 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,611 [shard 0] osd - peering_event(id=397, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): submitting ctx 2022-03-29T17:52:01.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,611 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15 for pg: 3.12 2022-03-29T17:52:01.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 unknown state: react 15 telling primary 2022-03-29T17:52:01.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - peering_event(id=399, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): submitting ctx 2022-03-29T17:52:01.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - dispatch_context_messages sending messages to 0 2022-03-29T17:52:01.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=14 evt=epoch_sent: 15 epoch_requested: 14 MLogRec from 0 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants= intervals=) pg_lease(ru 0.000000000s ub 141.971359253s int 16.000000000s) +create_info)): complete 2022-03-29T17:52:01.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,613 [shard 0] osd - handle_peering_op on 3.1c from 0 2022-03-29T17:52:01.643 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,613 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.014877319s))): start 2022-03-29T17:52:01.644 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,614 [shard 0] osd - peering_event(id=398, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.644 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,614 [shard 0] osd - peering_event(id=397, detail=PeeringEvent(from=1 pgid=3.12 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 IntervalFlush)): complete 2022-03-29T17:52:01.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - handle_peering_op on 3.0 from 2 2022-03-29T17:52:01.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.028182983s))): start 2022-03-29T17:52:01.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.014877319s))): got map 15 2022-03-29T17:52:01.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - peering_event(id=399, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 ActivateCommitted from 15 processed at 15)): complete 2022-03-29T17:52:01.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.028182983s))): got map 15 2022-03-29T17:52:01.646 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.014877319s))): pg present 2022-03-29T17:52:01.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.028182983s))): pg present 2022-03-29T17:52:01.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.014877319s) for pg: 3.1c 2022-03-29T17:52:01.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.014877319s 2022-03-29T17:52:01.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.014877319s (sent 142.014877319s) 2022-03-29T17:52:01.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:52:01.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.014877319s 2022-03-29T17:52:01.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.014877319s (sent 142.314071655s) 2022-03-29T17:52:01.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.314865112s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - peering_event(id=402, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.014877319s))): submitting ctx 2022-03-29T17:52:01.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - peering_event(id=402, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.028182983s) for pg: 3.0 2022-03-29T17:52:01.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.028182983s 2022-03-29T17:52:01.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.028182983s (sent 142.028182983s) 2022-03-29T17:52:01.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:01.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.028182983s 2022-03-29T17:52:01.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.667 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.667 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.667 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,590 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.667 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.956466675s))): submitting ctx 2022-03-29T17:52:01.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,591 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): pg present 2022-03-29T17:52:01.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,593 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): got map 15 2022-03-29T17:52:01.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): pg present 2022-03-29T17:52:01.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,594 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s) for pg: 3.3 2022-03-29T17:52:01.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,595 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,596 [shard 0] osd - peering_event(id=190, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.956466675s))): complete 2022-03-29T17:52:01.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,596 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.11 2022-03-29T17:52:01.674 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.271438 3 0.007263 2022-03-29T17:52:01.674 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,597 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3255389, 0.007263504 spent on 3 events 2022-03-29T17:52:01.675 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.675 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,597 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:01.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.679 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,597 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000234 0 0.000000 2022-03-29T17:52:01.679 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,598 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.5970306, 0.0 spent on 0 events 2022-03-29T17:52:01.679 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.680 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,598 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,598 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s) for pg: 3.19 2022-03-29T17:52:01.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,599 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,600 [shard 0] osd - peering_event(id=191, detail=PeeringEvent(from=0 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.975814819s ub 142.146469116s int 16.000000000s))): complete 2022-03-29T17:52:01.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - handle_peering_op on 3.1d from 0 2022-03-29T17:52:01.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.095642090s))): start 2022-03-29T17:52:01.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - peering_event(id=193, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,601 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.095642090s))): got map 15 2022-03-29T17:52:01.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.095642090s))): pg present 2022-03-29T17:52:01.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,602 [shard 0] osd - peering_event(id=192, detail=PeeringEvent(from=1 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.007873535s ub 142.247009277s int 16.000000000s))): complete 2022-03-29T17:52:01.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.095642090s) for pg: 3.1d 2022-03-29T17:52:01.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.095642090s 2022-03-29T17:52:01.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.095642090s (sent 142.095642090s) 2022-03-29T17:52:01.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.095642090s))): submitting ctx 2022-03-29T17:52:01.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,603 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - peering_event(id=194, detail=PeeringEvent(from=0 pgid=3.1d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.095642090s))): complete 2022-03-29T17:52:01.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,604 [shard 0] osd - handle_peering_op on 3.18 from 0 2022-03-29T17:52:01.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,605 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.121215820s))): start 2022-03-29T17:52:01.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,605 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.121215820s))): got map 15 2022-03-29T17:52:01.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,605 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.121215820s))): pg present 2022-03-29T17:52:01.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.121215820s) for pg: 3.18 2022-03-29T17:52:01.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.121215820s 2022-03-29T17:52:01.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.121215820s (sent 142.121215820s) 2022-03-29T17:52:01.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.121215820s))): submitting ctx 2022-03-29T17:52:01.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,606 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:52:01.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): start 2022-03-29T17:52:01.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - peering_event(id=195, detail=PeeringEvent(from=0 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.121215820s))): complete 2022-03-29T17:52:01.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): got map 15 2022-03-29T17:52:01.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,608 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): pg present 2022-03-29T17:52:01.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s) for pg: 3.14 2022-03-29T17:52:01.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s) 2022-03-29T17:52:01.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.242950439s 2022-03-29T17:52:01.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,609 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.242950439s 2022-03-29T17:52:01.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.693 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.693 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.693 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:52:01.693 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): start 2022-03-29T17:52:01.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,611 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): got map 15 2022-03-29T17:52:01.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,611 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): pg present 2022-03-29T17:52:01.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,611 [shard 0] osd - peering_event(id=196, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): complete 2022-03-29T17:52:01.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s) for pg: 3.14 2022-03-29T17:52:01.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,613 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,613 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,613 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:52:01.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,613 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): start 2022-03-29T17:52:01.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,613 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): got map 15 2022-03-29T17:52:01.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,614 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): pg present 2022-03-29T17:52:01.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - peering_event(id=197, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): complete 2022-03-29T17:52:01.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s) for pg: 3.14 2022-03-29T17:52:01.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - peering_event(id=198, detail=PeeringEvent(from=1 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.024856567s ub 142.294967651s int 16.000000000s))): complete 2022-03-29T17:52:01.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,625 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:52:01.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,625 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): start 2022-03-29T17:52:01.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,625 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): got map 15 2022-03-29T17:52:01.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,625 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): pg present 2022-03-29T17:52:01.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:52:01.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): start 2022-03-29T17:52:01.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s) for pg: 3.0 2022-03-29T17:52:01.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s) 2022-03-29T17:52:01.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.259384155s 2022-03-29T17:52:01.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.259384155s 2022-03-29T17:52:01.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): got map 15 2022-03-29T17:52:01.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): pg present 2022-03-29T17:52:01.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s) for pg: 3.0 2022-03-29T17:52:01.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,626 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - peering_event(id=199, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): complete 2022-03-29T17:52:01.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - peering_event(id=200, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): complete 2022-03-29T17:52:01.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:52:01.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): start 2022-03-29T17:52:01.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): got map 15 2022-03-29T17:52:01.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): pg present 2022-03-29T17:52:01.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s) for pg: 3.0 2022-03-29T17:52:01.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.706 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,610 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=2 pgid=3.18 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.936660767s ub 142.121215820s int 16.000000000s))): complete 2022-03-29T17:52:01.706 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,611 [shard 0] osd - handle_peering_op on 3.a from 1 2022-03-29T17:52:01.706 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,611 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.957992554s))): start 2022-03-29T17:52:01.706 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.091354370s))): got map 15 2022-03-29T17:52:01.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.957992554s))): got map 15 2022-03-29T17:52:01.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,612 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.091354370s))): pg present 2022-03-29T17:52:01.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,613 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.957992554s))): pg present 2022-03-29T17:52:01.707 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,614 [shard 0] osd - handle_peering_op on 3.c from 1 2022-03-29T17:52:01.708 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,614 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.962356567s))): start 2022-03-29T17:52:01.708 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,614 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.957992554s) for pg: 3.a 2022-03-29T17:52:01.708 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.957992554s 2022-03-29T17:52:01.708 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.957992554s (sent 141.957992554s) 2022-03-29T17:52:01.709 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.709 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.709 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.709 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.709 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,615 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.710 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.710 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.710 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.710 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.711 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.711 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.711 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.711 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,616 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.957992554s 2022-03-29T17:52:01.711 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.957992554s (sent 142.225738525s) 2022-03-29T17:52:01.712 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.712 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.712 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.712 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.713 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.713 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.713 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.713 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.714 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.226333618s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.714 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.714 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.714 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.715 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.715 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.715 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.715 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.716 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.716 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.716 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.716 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.716 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,617 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.717 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.717 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.717 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.717 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.718 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.718 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.718 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.718 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.719 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.719 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.719 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.719 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.719 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.720 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.720 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.720 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.957992554s))): submitting ctx 2022-03-29T17:52:01.720 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.091354370s) for pg: 3.9 2022-03-29T17:52:01.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.091354370s 2022-03-29T17:52:01.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,618 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.091354370s (sent 142.091354370s) 2022-03-29T17:52:01.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.721 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.091354370s))): submitting ctx 2022-03-29T17:52:01.722 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.722 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.722 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,619 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.962356567s))): got map 15 2022-03-29T17:52:01.722 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.962356567s))): pg present 2022-03-29T17:52:01.723 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,620 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.9 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.091354370s))): complete 2022-03-29T17:52:01.723 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.a 2022-03-29T17:52:01.723 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.270682 4 0.006785 2022-03-29T17:52:01.723 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,621 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.350692, 0.006785817 spent on 4 events 2022-03-29T17:52:01.724 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.724 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,621 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.724 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.725 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.726 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.726 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000236 0 0.000000 2022-03-29T17:52:01.727 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,621 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6214368, 0.0 spent on 0 events 2022-03-29T17:52:01.727 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.727 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,621 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.727 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,621 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.728 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - handle_peering_op on 3.1f from 1 2022-03-29T17:52:01.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.966506958s))): start 2022-03-29T17:52:01.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.957992554s))): complete 2022-03-29T17:52:01.729 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.962356567s) for pg: 3.c 2022-03-29T17:52:01.730 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.962356567s 2022-03-29T17:52:01.730 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.962356567s (sent 141.962356567s) 2022-03-29T17:52:01.730 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.730 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.731 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.732 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.732 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.732 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.732 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.962356567s 2022-03-29T17:52:01.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.962356567s (sent 142.233489990s) 2022-03-29T17:52:01.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.733 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.734 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.734 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.734 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.734 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.236778259s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.028182983s (sent 142.317977905s) 2022-03-29T17:52:01.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.318778992s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,622 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - peering_event(id=403, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,623 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.028182983s))): submitting ctx 2022-03-29T17:52:01.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - handle_peering_op on 3.1a from 2 2022-03-29T17:52:01.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,624 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.033203125s))): start 2022-03-29T17:52:01.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,625 [shard 0] osd - peering_event(id=403, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,627 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.014877319s))): complete 2022-03-29T17:52:01.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - handle_peering_op on 3.b from 0 2022-03-29T17:52:01.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.037933350s))): start 2022-03-29T17:52:01.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.1c 2022-03-29T17:52:01.751 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.307563 4 0.006200 2022-03-29T17:52:01.751 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,628 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3211188, 0.006200272 spent on 4 events 2022-03-29T17:52:01.751 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.751 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,628 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.754 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.754 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000703 0 0.000000 2022-03-29T17:52:01.754 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,629 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6288402, 0.0 spent on 0 events 2022-03-29T17:52:01.755 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.755 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,629 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.755 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,630 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,630 [shard 0] osd - peering_event(id=402, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,630 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.756 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,630 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.033203125s))): got map 15 2022-03-29T17:52:01.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.028182983s))): complete 2022-03-29T17:52:01.757 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.0 2022-03-29T17:52:01.757 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.296491 4 0.005795 2022-03-29T17:52:01.757 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.334877, 0.005795217 spent on 4 events 2022-03-29T17:52:01.757 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.758 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.758 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.758 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.758 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.759 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.759 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.759 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.759 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.759 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.760 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.760 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.760 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.760 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000243 0 0.000000 2022-03-29T17:52:01.761 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6314316, 0.0 spent on 0 events 2022-03-29T17:52:01.761 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.761 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.761 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.761 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - peering_event(id=403, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.762 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.763 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.033203125s))): pg present 2022-03-29T17:52:01.763 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.037933350s))): got map 15 2022-03-29T17:52:01.763 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,634 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.037933350s))): pg present 2022-03-29T17:52:01.763 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - handle_peering_op on 3.13 from 0 2022-03-29T17:52:01.764 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.042297363s))): start 2022-03-29T17:52:01.764 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - peering_event(id=402, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.764 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.033203125s) for pg: 3.1a 2022-03-29T17:52:01.764 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.033203125s 2022-03-29T17:52:01.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.033203125s (sent 142.033203125s) 2022-03-29T17:52:01.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:01.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.765 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.766 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.033203125s 2022-03-29T17:52:01.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.033203125s (sent 142.332717896s) 2022-03-29T17:52:01.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.332984924s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.770 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.771 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.772 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.773 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - peering_event(id=407, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.774 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,633 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,634 [shard 0] osd - peering_event(id=201, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.028182983s ub 142.317977905s int 16.000000000s))): complete 2022-03-29T17:52:01.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,644 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:52:01.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,644 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): start 2022-03-29T17:52:01.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,644 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): got map 15 2022-03-29T17:52:01.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,644 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): pg present 2022-03-29T17:52:01.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:52:01.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): start 2022-03-29T17:52:01.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s) for pg: 3.1a 2022-03-29T17:52:01.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s) 2022-03-29T17:52:01.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.278366089s 2022-03-29T17:52:01.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.278366089s 2022-03-29T17:52:01.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): got map 15 2022-03-29T17:52:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): pg present 2022-03-29T17:52:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s) for pg: 3.1a 2022-03-29T17:52:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.780 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,645 [shard 0] osd - peering_event(id=202, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): complete 2022-03-29T17:52:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - peering_event(id=203, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): complete 2022-03-29T17:52:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:52:01.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): start 2022-03-29T17:52:01.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): got map 15 2022-03-29T17:52:01.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): pg present 2022-03-29T17:52:01.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s) for pg: 3.1a 2022-03-29T17:52:01.782 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.783 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.783 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.783 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.783 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - peering_event(id=204, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.033203125s ub 142.332717896s int 16.000000000s))): complete 2022-03-29T17:52:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,661 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:52:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,661 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): start 2022-03-29T17:52:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,661 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): got map 15 2022-03-29T17:52:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,661 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): pg present 2022-03-29T17:52:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:52:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): start 2022-03-29T17:52:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s) for pg: 3.2 2022-03-29T17:52:01.785 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s) 2022-03-29T17:52:01.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.295455933s 2022-03-29T17:52:01.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.295455933s 2022-03-29T17:52:01.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.786 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): got map 15 2022-03-29T17:52:01.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): pg present 2022-03-29T17:52:01.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s) for pg: 3.2 2022-03-29T17:52:01.787 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,662 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.788 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,663 [shard 0] osd - peering_event(id=205, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): complete 2022-03-29T17:52:01.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - peering_event(id=206, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): complete 2022-03-29T17:52:01.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:52:01.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): start 2022-03-29T17:52:01.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): got map 15 2022-03-29T17:52:01.789 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): pg present 2022-03-29T17:52:01.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s) for pg: 3.2 2022-03-29T17:52:01.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.790 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,669 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,669 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,669 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,669 [shard 0] osd - peering_event(id=207, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.047134399s ub 142.353500366s int 16.000000000s))): complete 2022-03-29T17:52:01.791 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,680 [shard 0] osd - handle_peering_op on 3.e from 0 2022-03-29T17:52:01.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,680 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.164947510s))): start 2022-03-29T17:52:01.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,680 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.164947510s))): got map 15 2022-03-29T17:52:01.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,680 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.164947510s))): pg present 2022-03-29T17:52:01.792 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,681 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.164947510s) for pg: 3.e 2022-03-29T17:52:01.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,681 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.164947510s 2022-03-29T17:52:01.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,681 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.164947510s (sent 142.164947510s) 2022-03-29T17:52:01.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,681 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,681 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.164947510s))): submitting ctx 2022-03-29T17:52:01.793 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,681 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,681 [shard 0] osd - peering_event(id=208, detail=PeeringEvent(from=0 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.164947510s))): complete 2022-03-29T17:52:01.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,682 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:52:01.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,682 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): start 2022-03-29T17:52:01.794 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,682 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): got map 15 2022-03-29T17:52:01.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,682 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): pg present 2022-03-29T17:52:01.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,682 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:52:01.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): start 2022-03-29T17:52:01.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s) for pg: 3.4 2022-03-29T17:52:01.795 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s) 2022-03-29T17:52:01.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.316268921s 2022-03-29T17:52:01.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.316268921s 2022-03-29T17:52:01.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.796 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): got map 15 2022-03-29T17:52:01.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): pg present 2022-03-29T17:52:01.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s) for pg: 3.4 2022-03-29T17:52:01.797 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.798 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,683 [shard 0] osd - peering_event(id=209, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): complete 2022-03-29T17:52:01.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - peering_event(id=210, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): complete 2022-03-29T17:52:01.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:52:01.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): start 2022-03-29T17:52:01.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): got map 15 2022-03-29T17:52:01.799 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): pg present 2022-03-29T17:52:01.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s) for pg: 3.4 2022-03-29T17:52:01.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.800 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,691 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,692 [shard 0] osd - peering_event(id=211, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.054275513s ub 142.370239258s int 16.000000000s))): complete 2022-03-29T17:52:01.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - handle_peering_op on 3.8 from 1 2022-03-29T17:52:01.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.110671997s))): start 2022-03-29T17:52:01.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.110671997s))): got map 15 2022-03-29T17:52:01.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.110671997s))): pg present 2022-03-29T17:52:01.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.110671997s) for pg: 3.8 2022-03-29T17:52:01.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.110671997s 2022-03-29T17:52:01.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.110671997s (sent 142.110671997s) 2022-03-29T17:52:01.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.110671997s))): submitting ctx 2022-03-29T17:52:01.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,699 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,700 [shard 0] osd - peering_event(id=212, detail=PeeringEvent(from=1 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.110671997s))): complete 2022-03-29T17:52:01.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,740 [shard 0] osd - handle_peering_op on 3.7 from 1 2022-03-29T17:52:01.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,740 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.129089355s))): start 2022-03-29T17:52:01.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,740 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.129089355s))): got map 15 2022-03-29T17:52:01.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,740 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.129089355s))): pg present 2022-03-29T17:52:01.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.129089355s) for pg: 3.7 2022-03-29T17:52:01.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.129089355s 2022-03-29T17:52:01.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.129089355s (sent 142.129089355s) 2022-03-29T17:52:01.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.129089355s))): submitting ctx 2022-03-29T17:52:01.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - peering_event(id=213, detail=PeeringEvent(from=1 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.129089355s))): complete 2022-03-29T17:52:01.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - handle_peering_op on 3.1e from 1 2022-03-29T17:52:01.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.150024414s))): start 2022-03-29T17:52:01.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.150024414s))): got map 15 2022-03-29T17:52:01.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,741 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.150024414s))): pg present 2022-03-29T17:52:01.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.150024414s) for pg: 3.1e 2022-03-29T17:52:01.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.150024414s 2022-03-29T17:52:01.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.150024414s (sent 142.150024414s) 2022-03-29T17:52:01.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.150024414s))): submitting ctx 2022-03-29T17:52:01.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - handle_peering_op on 3.16 from 1 2022-03-29T17:52:01.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,628 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.962356567s))): submitting ctx 2022-03-29T17:52:01.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,629 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,630 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.966506958s))): got map 15 2022-03-29T17:52:01.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,630 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,630 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.966506958s))): pg present 2022-03-29T17:52:01.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - handle_peering_op on 3.12 from 1 2022-03-29T17:52:01.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.971359253s))): start 2022-03-29T17:52:01.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.962356567s))): complete 2022-03-29T17:52:01.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.c 2022-03-29T17:52:01.813 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.276807 4 0.008448 2022-03-29T17:52:01.813 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.355073, 0.008448769 spent on 4 events 2022-03-29T17:52:01.814 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.814 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,631 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,631 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.817 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000230 0 0.000000 2022-03-29T17:52:01.817 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,632 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6319342, 0.0 spent on 0 events 2022-03-29T17:52:01.817 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.817 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,632 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,632 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,634 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,634 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,634 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,634 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,634 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.971359253s))): got map 15 2022-03-29T17:52:01.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.971359253s))): pg present 2022-03-29T17:52:01.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.966506958s) for pg: 3.1f 2022-03-29T17:52:01.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.966506958s 2022-03-29T17:52:01.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.966506958s (sent 141.966506958s) 2022-03-29T17:52:01.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.821 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,635 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.821 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.821 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.821 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.822 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.966506958s 2022-03-29T17:52:01.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.966506958s (sent 142.245391846s) 2022-03-29T17:52:01.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,636 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.246192932s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,637 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.966506958s))): submitting ctx 2022-03-29T17:52:01.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - handle_peering_op on 3.e from 2 2022-03-29T17:52:01.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): start 2022-03-29T17:52:01.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,639 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.966506958s))): complete 2022-03-29T17:52:01.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): got map 15 2022-03-29T17:52:01.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): pg present 2022-03-29T17:52:01.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.971359253s) for pg: 3.12 2022-03-29T17:52:01.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.971359253s 2022-03-29T17:52:01.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.971359253s (sent 141.971359253s) 2022-03-29T17:52:01.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.1 activated and committed 2022-03-29T17:52:01.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.1 ruub 141.971359253s 2022-03-29T17:52:01.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 141.971359253s (sent 142.250366211s) 2022-03-29T17:52:01.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.250610352s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.033203125s))): submitting ctx 2022-03-29T17:52:01.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,638 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.042297363s))): got map 15 2022-03-29T17:52:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,639 [shard 0] osd - handle_peering_op on 3.2 from 2 2022-03-29T17:52:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,639 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.047134399s))): start 2022-03-29T17:52:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,639 [shard 0] osd - peering_event(id=407, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:01.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.037933350s) for pg: 3.b 2022-03-29T17:52:01.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.037933350s 2022-03-29T17:52:01.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.037933350s (sent 142.037933350s) 2022-03-29T17:52:01.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:52:01.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,640 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.037933350s 2022-03-29T17:52:01.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.037933350s (sent 142.336990356s) 2022-03-29T17:52:01.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.850 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.337776184s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,641 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - peering_event(id=409, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.037933350s))): submitting ctx 2022-03-29T17:52:01.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.042297363s))): pg present 2022-03-29T17:52:01.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - client_request(id=141, detail=m=[osd_op(client.4179.0:2 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e15) v8]): got obc lock 2022-03-29T17:52:01.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:01.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,644 [shard 0] osd - peering_event(id=409, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - peering_event(id=403, detail=PeeringEvent(from=1 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - with_head_obc: cache miss on 3:5cea7035:::rbd_info:head 2022-03-29T17:52:01.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.1a 2022-03-29T17:52:01.859 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.308248 4 0.006019 2022-03-29T17:52:01.860 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3388941, 0.006019799 spent on 4 events 2022-03-29T17:52:01.860 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.860 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000773 0 0.000000 2022-03-29T17:52:01.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6471894, 0.0 spent on 0 events 2022-03-29T17:52:01.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,648 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,648 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - peering_event(id=407, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.033203125s))): complete 2022-03-29T17:52:01.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.047134399s))): got map 15 2022-03-29T17:52:01.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,649 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.042297363s) for pg: 3.13 2022-03-29T17:52:01.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.042297363s 2022-03-29T17:52:01.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.042297363s (sent 142.042297363s) 2022-03-29T17:52:01.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:52:01.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.042297363s 2022-03-29T17:52:01.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.042297363s (sent 142.347534180s) 2022-03-29T17:52:01.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.348320007s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - peering_event(id=410, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.175064087s))): start 2022-03-29T17:52:01.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.175064087s))): got map 15 2022-03-29T17:52:01.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - peering_event(id=214, detail=PeeringEvent(from=1 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.150024414s))): complete 2022-03-29T17:52:01.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.175064087s))): pg present 2022-03-29T17:52:01.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.175064087s) for pg: 3.16 2022-03-29T17:52:01.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.175064087s 2022-03-29T17:52:01.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,742 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.175064087s (sent 142.175064087s) 2022-03-29T17:52:01.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.175064087s))): submitting ctx 2022-03-29T17:52:01.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - handle_peering_op on 3.5 from 1 2022-03-29T17:52:01.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.193099976s))): start 2022-03-29T17:52:01.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - peering_event(id=215, detail=PeeringEvent(from=1 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.175064087s))): complete 2022-03-29T17:52:01.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.193099976s))): got map 15 2022-03-29T17:52:01.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.193099976s))): pg present 2022-03-29T17:52:01.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.193099976s) for pg: 3.5 2022-03-29T17:52:01.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.193099976s 2022-03-29T17:52:01.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,743 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.193099976s (sent 142.193099976s) 2022-03-29T17:52:01.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.193099976s))): submitting ctx 2022-03-29T17:52:01.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - handle_peering_op on 3.11 from 1 2022-03-29T17:52:01.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.221298218s))): start 2022-03-29T17:52:01.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - peering_event(id=216, detail=PeeringEvent(from=1 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.193099976s))): complete 2022-03-29T17:52:01.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.221298218s))): got map 15 2022-03-29T17:52:01.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.221298218s))): pg present 2022-03-29T17:52:01.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.221298218s) for pg: 3.11 2022-03-29T17:52:01.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.221298218s 2022-03-29T17:52:01.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.221298218s (sent 142.221298218s) 2022-03-29T17:52:01.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.221298218s))): submitting ctx 2022-03-29T17:52:01.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,744 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:51:59,745 [shard 0] osd - peering_event(id=217, detail=PeeringEvent(from=1 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.221298218s))): complete 2022-03-29T17:52:01.884 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:51:59,820 [shard 0] alienstore - stat 2022-03-29T17:52:01.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,642 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.971359253s))): submitting ctx 2022-03-29T17:52:01.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,643 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.1f 2022-03-29T17:52:01.889 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.287060 4 0.005500 2022-03-29T17:52:01.889 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,646 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3592374, 0.005500468 spent on 4 events 2022-03-29T17:52:01.889 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.889 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,646 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.890 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.890 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.890 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.890 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.890 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.891 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.891 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.891 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,646 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.895 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.896 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.896 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000987 0 0.000000 2022-03-29T17:52:01.896 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6463606, 0.0 spent on 0 events 2022-03-29T17:52:01.896 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.897 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,647 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.897 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,647 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - handle_peering_op on 3.e from 2 2022-03-29T17:52:01.898 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,648 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): start 2022-03-29T17:52:01.899 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 141.971359253s))): complete 2022-03-29T17:52:01.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s) for pg: 3.e 2022-03-29T17:52:01.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s) 2022-03-29T17:52:01.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.259582520s 2022-03-29T17:52:01.900 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,650 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.259582520s 2022-03-29T17:52:01.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): got map 15 2022-03-29T17:52:01.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - handle_peering_op on 3.1c from 1 2022-03-29T17:52:01.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,651 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): start 2022-03-29T17:52:01.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.12 2022-03-29T17:52:01.902 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.288012 4 0.005653 2022-03-29T17:52:01.902 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,652 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3640635, 0.005653919 spent on 4 events 2022-03-29T17:52:01.903 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.903 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,652 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 primary objects 0 missing 0 2022-03-29T17:52:01.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 objects 0 missing 0 2022-03-29T17:52:01.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.907 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000764 0 0.000000 2022-03-29T17:52:01.907 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,652 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6521225, 0.0 spent on 0 events 2022-03-29T17:52:01.908 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.908 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,652 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,652 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): pg present 2022-03-29T17:52:01.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): got map 15 2022-03-29T17:52:01.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): complete 2022-03-29T17:52:01.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): pg present 2022-03-29T17:52:01.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s) for pg: 3.e 2022-03-29T17:52:01.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - handle_peering_op on 3.e from 2 2022-03-29T17:52:01.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): start 2022-03-29T17:52:01.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s) for pg: 3.1c 2022-03-29T17:52:01.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s) 2022-03-29T17:52:01.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.264923096s 2022-03-29T17:52:01.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.264923096s 2022-03-29T17:52:01.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - handle_peering_op on 3.1c from 1 2022-03-29T17:52:01.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): start 2022-03-29T17:52:01.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): got map 15 2022-03-29T17:52:01.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): pg present 2022-03-29T17:52:01.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): complete 2022-03-29T17:52:01.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): got map 15 2022-03-29T17:52:01.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s) for pg: 3.e 2022-03-29T17:52:01.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): pg present 2022-03-29T17:52:01.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): complete 2022-03-29T17:52:01.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s) for pg: 3.1c 2022-03-29T17:52:01.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,660 [shard 0] osd - handle_peering_op on 3.1c from 1 2022-03-29T17:52:01.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,660 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): start 2022-03-29T17:52:01.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,660 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): got map 15 2022-03-29T17:52:01.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,661 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): pg present 2022-03-29T17:52:01.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,661 [shard 0] osd - handle_peering_op on 3.1 from 2 2022-03-29T17:52:01.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,661 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.127700806s))): start 2022-03-29T17:52:01.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): complete 2022-03-29T17:52:01.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=2 pgid=3.e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.963928223s ub 142.164947510s int 16.000000000s))): complete 2022-03-29T17:52:01.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s) for pg: 3.1c 2022-03-29T17:52:01.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.127700806s))): got map 15 2022-03-29T17:52:01.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.127700806s))): pg present 2022-03-29T17:52:01.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.127700806s) for pg: 3.1 2022-03-29T17:52:01.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.127700806s 2022-03-29T17:52:01.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.127700806s (sent 142.127700806s) 2022-03-29T17:52:01.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.127700806s))): submitting ctx 2022-03-29T17:52:01.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - handle_peering_op on 3.b from 1 2022-03-29T17:52:01.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): start 2022-03-29T17:52:01.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=1 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.014877319s ub 142.314071655s int 16.000000000s))): complete 2022-03-29T17:52:01.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): got map 15 2022-03-29T17:52:01.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): pg present 2022-03-29T17:52:01.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,669 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s) for pg: 3.b 2022-03-29T17:52:01.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s) 2022-03-29T17:52:01.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.279022217s 2022-03-29T17:52:01.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.279022217s 2022-03-29T17:52:01.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.042297363s))): submitting ctx 2022-03-29T17:52:01.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,653 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.047134399s))): pg present 2022-03-29T17:52:01.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.b 2022-03-29T17:52:01.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.310932 4 0.005498 2022-03-29T17:52:01.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,654 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3435788, 0.00549802 spent on 4 events 2022-03-29T17:52:01.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,654 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,654 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.932 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,655 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000734 0 0.000000 2022-03-29T17:52:01.932 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,655 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6545582, 0.0 spent on 0 events 2022-03-29T17:52:01.933 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,655 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.933 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,655 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - peering_event(id=409, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.037933350s))): complete 2022-03-29T17:52:01.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,655 [shard 0] osd - load_metadata: object 3:5cea7035:::rbd_info:head doesn't exist, returning empty metadata 2022-03-29T17:52:01.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - load_head_obc: loaded obs 3:5cea7035:::rbd_info:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:5cea7035:::rbd_info:head 2022-03-29T17:52:01.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - load_head_obc: returning obc 3:5cea7035:::rbd_info:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:5cea7035:::rbd_info:head 2022-03-29T17:52:01.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - peering_event(id=407, detail=PeeringEvent(from=1 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - peering_event(id=410, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,656 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.047134399s) for pg: 3.2 2022-03-29T17:52:01.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.047134399s 2022-03-29T17:52:01.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.047134399s (sent 142.047134399s) 2022-03-29T17:52:01.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:01.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:01.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:01.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.047134399s 2022-03-29T17:52:01.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.047134399s (sent 142.353500366s) 2022-03-29T17:52:01.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,657 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:01.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:01.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:01.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.354293823s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:01.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - send_pg_created 2022-03-29T17:52:01.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:01.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:01.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:01.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:01.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - peering_event(id=411, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:01.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,658 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:01.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.047134399s))): submitting ctx 2022-03-29T17:52:01.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op call 2022-03-29T17:52:01.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,659 [shard 0] osd - handling op call on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:01.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,660 [shard 0] osd - rollback_obc_if_modified: object 3:5cea7035:::rbd_info:head got error generic:2, need_rollback=false 2022-03-29T17:52:01.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,660 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.042297363s))): complete 2022-03-29T17:52:01.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,660 [shard 0] osd - peering_event(id=411, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:01.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,663 [shard 0] osd - handle_peering_op on 3.4 from 2 2022-03-29T17:52:01.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,663 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.054275513s))): start 2022-03-29T17:52:01.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,664 [shard 0] osd - peering_event(id=409, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.2 2022-03-29T17:52:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.311230 4 0.005653 2022-03-29T17:52:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,665 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3538475, 0.00565346 spent on 4 events 2022-03-29T17:52:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,665 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:01.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:01.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,665 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:01.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000956 0 0.000000 2022-03-29T17:52:01.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,666 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6651492, 0.0 spent on 0 events 2022-03-29T17:52:01.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,666 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,666 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - peering_event(id=411, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:01.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - handle_peering_op on 3.d from 0 2022-03-29T17:52:01.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.059051514s))): start 2022-03-29T17:52:01.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,667 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.054275513s))): got map 15 2022-03-29T17:52:01.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:01.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.047134399s))): complete 2022-03-29T17:52:01.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,668 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.13 2022-03-29T17:52:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.322326 4 0.007273 2022-03-29T17:52:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,670 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.34797, 0.007273984 spent on 4 events 2022-03-29T17:52:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,670 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:01.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:01.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:01.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:01.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:01.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:01.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:01.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:01.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:01.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:01.960 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000246 0 0.000000 2022-03-29T17:52:01.960 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,670 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.6703558, 0.0 spent on 0 events 2022-03-29T17:52:01.960 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:01.961 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,670 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:01.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:01.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:01.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:01.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=3.1 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.127700806s))): complete 2022-03-29T17:52:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,670 [shard 0] osd - handle_peering_op on 3.b from 1 2022-03-29T17:52:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): start 2022-03-29T17:52:01.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): got map 15 2022-03-29T17:52:01.964 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): pg present 2022-03-29T17:52:01.964 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): complete 2022-03-29T17:52:01.964 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - handle_peering_op on 3.3 from 2 2022-03-29T17:52:01.964 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.146469116s))): start 2022-03-29T17:52:01.965 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - handle_peering_op on 3.13 from 1 2022-03-29T17:52:01.965 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): start 2022-03-29T17:52:01.965 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s) for pg: 3.b 2022-03-29T17:52:01.965 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.965 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.966 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.966 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.966 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.146469116s))): got map 15 2022-03-29T17:52:01.966 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): got map 15 2022-03-29T17:52:01.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.146469116s))): pg present 2022-03-29T17:52:01.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - handle_peering_op on 3.13 from 1 2022-03-29T17:52:01.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): start 2022-03-29T17:52:01.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): pg present 2022-03-29T17:52:01.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): complete 2022-03-29T17:52:01.975 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): got map 15 2022-03-29T17:52:01.975 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.146469116s) for pg: 3.3 2022-03-29T17:52:01.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.146469116s 2022-03-29T17:52:01.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.146469116s (sent 142.146469116s) 2022-03-29T17:52:01.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.977 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.146469116s))): submitting ctx 2022-03-29T17:52:01.977 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.978 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): pg present 2022-03-29T17:52:01.978 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s) for pg: 3.13 2022-03-29T17:52:01.979 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s) 2022-03-29T17:52:01.979 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.285461426s 2022-03-29T17:52:01.980 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.285461426s 2022-03-29T17:52:01.980 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.981 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.981 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.982 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - handle_peering_op on 3.b from 1 2022-03-29T17:52:01.982 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): start 2022-03-29T17:52:01.982 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s) for pg: 3.13 2022-03-29T17:52:01.983 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:01.983 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.984 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.984 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.985 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): got map 15 2022-03-29T17:52:01.985 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.3 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.146469116s))): complete 2022-03-29T17:52:01.986 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): pg present 2022-03-29T17:52:01.986 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s) for pg: 3.b 2022-03-29T17:52:01.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): complete 2022-03-29T17:52:01.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - handle_peering_op on 3.13 from 1 2022-03-29T17:52:01.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,681 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): start 2022-03-29T17:52:01.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,684 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): complete 2022-03-29T17:52:01.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,684 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=1 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.037933350s ub 142.336990356s int 16.000000000s))): complete 2022-03-29T17:52:01.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): got map 15 2022-03-29T17:52:01.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): pg present 2022-03-29T17:52:01.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - handle_peering_op on 3.d from 1 2022-03-29T17:52:01.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): start 2022-03-29T17:52:01.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s) for pg: 3.13 2022-03-29T17:52:01.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:01.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:01.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - log is not dirty 2022-03-29T17:52:01.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): submitting ctx 2022-03-29T17:52:01.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:01.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,687 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): got map 15 2022-03-29T17:52:01.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,687 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): pg present 2022-03-29T17:52:01.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,687 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.042297363s ub 142.347534180s int 16.000000000s))): complete 2022-03-29T17:52:02.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,687 [shard 0] osd - handle_peering_op on 3.d from 1 2022-03-29T17:52:02.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,687 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): start 2022-03-29T17:52:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,687 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): got map 15 2022-03-29T17:52:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,688 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): pg present 2022-03-29T17:52:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,688 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s) for pg: 3.d 2022-03-29T17:52:02.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,688 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s) 2022-03-29T17:52:02.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,688 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.297821045s 2022-03-29T17:52:02.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,688 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.297821045s 2022-03-29T17:52:02.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s) for pg: 3.d 2022-03-29T17:52:02.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): complete 2022-03-29T17:52:02.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,692 [shard 0] osd - handle_peering_op on 3.d from 1 2022-03-29T17:52:02.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,692 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): start 2022-03-29T17:52:02.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,692 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): got map 15 2022-03-29T17:52:02.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,692 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): pg present 2022-03-29T17:52:02.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): complete 2022-03-29T17:52:02.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s) for pg: 3.d 2022-03-29T17:52:02.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.059051514s ub 142.373336792s int 16.000000000s))): complete 2022-03-29T17:52:02.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - handle_peering_op on 3.10 from 1 2022-03-29T17:52:02.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): start 2022-03-29T17:52:02.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): got map 15 2022-03-29T17:52:02.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - handle_peering_op on 3.10 from 1 2022-03-29T17:52:02.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): start 2022-03-29T17:52:02.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): pg present 2022-03-29T17:52:02.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): got map 15 2022-03-29T17:52:02.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s) for pg: 3.10 2022-03-29T17:52:02.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s) 2022-03-29T17:52:02.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.305603027s 2022-03-29T17:52:02.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.305603027s 2022-03-29T17:52:02.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,696 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): pg present 2022-03-29T17:52:02.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s) for pg: 3.10 2022-03-29T17:52:02.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): complete 2022-03-29T17:52:02.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.1 pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): complete 2022-03-29T17:52:02.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - handle_peering_op on 3.10 from 1 2022-03-29T17:52:02.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): start 2022-03-29T17:52:02.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): got map 15 2022-03-29T17:52:02.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): pg present 2022-03-29T17:52:02.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s) for pg: 3.10 2022-03-29T17:52:02.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,703 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,703 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,703 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 1 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 142.062164307s ub 142.389938354s int 16.000000000s))): complete 2022-03-29T17:52:02.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - handle_peering_op on 3.15 from 1 2022-03-29T17:52:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.137985229s))): start 2022-03-29T17:52:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,705 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.137985229s))): got map 15 2022-03-29T17:52:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,705 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.137985229s))): pg present 2022-03-29T17:52:02.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,705 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.137985229s) for pg: 3.15 2022-03-29T17:52:02.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,705 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.137985229s 2022-03-29T17:52:02.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - peering_event(id=410, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:02.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.054275513s))): pg present 2022-03-29T17:52:02.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,671 [shard 0] osd - OpSequencer::finish_op_in_order: op=141, last_started=141, last_unblocked=141, last_completed=0 2022-03-29T17:52:02.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.059051514s))): got map 15 2022-03-29T17:52:02.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - client_request(id=141, detail=m=[osd_op(client.4179.0:2 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e15) v8]): destroying 2022-03-29T17:52:02.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - peering_event(id=411, detail=PeeringEvent(from=1 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:02.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,672 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.059051514s))): pg present 2022-03-29T17:52:02.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - peering_event(id=410, detail=PeeringEvent(from=1 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:02.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - client_request(id=142, detail=m=[osd_op(client.4179.0:3 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e15) v8]): start 2022-03-29T17:52:02.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - client_request(id=142, detail=m=[osd_op(client.4179.0:3 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e15) v8]): in repeat 2022-03-29T17:52:02.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.054275513s) for pg: 3.4 2022-03-29T17:52:02.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.054275513s 2022-03-29T17:52:02.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.054275513s (sent 142.054275513s) 2022-03-29T17:52:02.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.2 activated and committed 2022-03-29T17:52:02.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:02.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:02.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:02.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:02.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:02.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:02.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,673 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:02.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:02.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.2 ruub 142.054275513s 2022-03-29T17:52:02.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.054275513s (sent 142.370239258s) 2022-03-29T17:52:02.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:02.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:02.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:02.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:02.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:02.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:02.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.370536804s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:02.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - send_pg_created 2022-03-29T17:52:02.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,674 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:02.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:02.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:02.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:02.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:02.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - peering_event(id=414, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:02.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,675 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:02.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.054275513s))): submitting ctx 2022-03-29T17:52:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.059051514s) for pg: 3.d 2022-03-29T17:52:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.059051514s 2022-03-29T17:52:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.059051514s (sent 142.059051514s) 2022-03-29T17:52:02.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:52:02.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:02.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:02.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,676 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:02.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:02.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:02.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.059051514s 2022-03-29T17:52:02.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.059051514s (sent 142.373336792s) 2022-03-29T17:52:02.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:02.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:02.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:02.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.373611450s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:02.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - send_pg_created 2022-03-29T17:52:02.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:02.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,677 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:02.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:02.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:02.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - peering_event(id=415, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:02.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:02.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.059051514s))): submitting ctx 2022-03-29T17:52:02.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,678 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - handle_peering_op on 3.10 from 0 2022-03-29T17:52:02.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,679 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.062164307s))): start 2022-03-29T17:52:02.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,680 [shard 0] osd - peering_event(id=415, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:02.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,680 [shard 0] osd - peering_event(id=414, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:02.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,684 [shard 0] osd - client_request(id=142, detail=m=[osd_op(client.4179.0:3 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e15) v8]) same_interval_since: 14 2022-03-29T17:52:02.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,684 [shard 0] osd - OpSequencer::start_op: op=142, last_started=141, last_unblocked=141, last_completed=141 2022-03-29T17:52:02.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.062164307s))): got map 15 2022-03-29T17:52:02.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.054275513s))): complete 2022-03-29T17:52:02.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.d 2022-03-29T17:52:02.059 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.320799 4 0.004329 2022-03-29T17:52:02.059 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,685 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3646967, 0.004329371 spent on 4 events 2022-03-29T17:52:02.059 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:02.059 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,685 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:02.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:02.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,685 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:02.062 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,686 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000766 0 0.000000 2022-03-29T17:52:02.062 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,686 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.685567, 0.0 spent on 0 events 2022-03-29T17:52:02.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,686 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:02.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,705 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.137985229s (sent 142.137985229s) 2022-03-29T17:52:02.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,705 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,705 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.137985229s))): submitting ctx 2022-03-29T17:52:02.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,705 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.065 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.137985229s))): complete 2022-03-29T17:52:02.065 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,750 [shard 0] osd - handle_peering_op on 3.17 from 1 2022-03-29T17:52:02.065 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,750 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.165069580s))): start 2022-03-29T17:52:02.065 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,750 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.165069580s))): got map 15 2022-03-29T17:52:02.066 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,750 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.165069580s))): pg present 2022-03-29T17:52:02.066 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.165069580s) for pg: 3.17 2022-03-29T17:52:02.066 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.165069580s 2022-03-29T17:52:02.066 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.165069580s (sent 142.165069580s) 2022-03-29T17:52:02.067 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.067 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.165069580s))): submitting ctx 2022-03-29T17:52:02.067 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.067 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - handle_peering_op on 3.6 from 1 2022-03-29T17:52:02.067 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.180969238s))): start 2022-03-29T17:52:02.068 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.180969238s))): got map 15 2022-03-29T17:52:02.068 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.165069580s))): complete 2022-03-29T17:52:02.068 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.180969238s))): pg present 2022-03-29T17:52:02.068 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.180969238s) for pg: 3.6 2022-03-29T17:52:02.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.180969238s 2022-03-29T17:52:02.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.180969238s (sent 142.180969238s) 2022-03-29T17:52:02.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,751 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,752 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.180969238s))): submitting ctx 2022-03-29T17:52:02.070 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,752 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.070 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,752 [shard 0] osd - handle_peering_op on 3.1b from 1 2022-03-29T17:52:02.070 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,752 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.197250366s))): start 2022-03-29T17:52:02.070 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,752 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.197250366s))): got map 15 2022-03-29T17:52:02.071 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,752 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.180969238s))): complete 2022-03-29T17:52:02.071 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,752 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.197250366s))): pg present 2022-03-29T17:52:02.071 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,752 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.197250366s) for pg: 3.1b 2022-03-29T17:52:02.071 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.197250366s 2022-03-29T17:52:02.071 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.197250366s (sent 142.197250366s) 2022-03-29T17:52:02.072 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.072 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.197250366s))): submitting ctx 2022-03-29T17:52:02.072 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.072 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - handle_peering_op on 3.f from 1 2022-03-29T17:52:02.072 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.209915161s))): start 2022-03-29T17:52:02.073 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=1 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.197250366s))): complete 2022-03-29T17:52:02.073 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.209915161s))): got map 15 2022-03-29T17:52:02.073 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.209915161s))): pg present 2022-03-29T17:52:02.073 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.209915161s) for pg: 3.f 2022-03-29T17:52:02.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.209915161s 2022-03-29T17:52:02.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.209915161s (sent 142.209915161s) 2022-03-29T17:52:02.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.209915161s))): submitting ctx 2022-03-29T17:52:02.074 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,753 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.075 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - handle_peering_op on 3.a from 1 2022-03-29T17:52:02.075 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.225738525s))): start 2022-03-29T17:52:02.075 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.225738525s))): got map 15 2022-03-29T17:52:02.075 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.225738525s))): pg present 2022-03-29T17:52:02.076 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.209915161s))): complete 2022-03-29T17:52:02.076 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.225738525s) for pg: 3.a 2022-03-29T17:52:02.076 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.225738525s 2022-03-29T17:52:02.076 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.225738525s (sent 142.225738525s) 2022-03-29T17:52:02.077 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.077 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.225738525s))): submitting ctx 2022-03-29T17:52:02.077 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,754 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.077 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - handle_peering_op on 3.c from 1 2022-03-29T17:52:02.077 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.233489990s))): start 2022-03-29T17:52:02.078 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.225738525s))): complete 2022-03-29T17:52:02.078 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.233489990s))): got map 15 2022-03-29T17:52:02.078 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.233489990s))): pg present 2022-03-29T17:52:02.078 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.233489990s) for pg: 3.c 2022-03-29T17:52:02.079 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.233489990s 2022-03-29T17:52:02.079 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.233489990s (sent 142.233489990s) 2022-03-29T17:52:02.079 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.079 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.233489990s))): submitting ctx 2022-03-29T17:52:02.079 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,755 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - handle_peering_op on 3.1f from 1 2022-03-29T17:52:02.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.245391846s))): start 2022-03-29T17:52:02.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.245391846s))): got map 15 2022-03-29T17:52:02.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.245391846s))): pg present 2022-03-29T17:52:02.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.233489990s))): complete 2022-03-29T17:52:02.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.245391846s) for pg: 3.1f 2022-03-29T17:52:02.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.245391846s 2022-03-29T17:52:02.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.245391846s (sent 142.245391846s) 2022-03-29T17:52:02.082 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.082 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.245391846s))): submitting ctx 2022-03-29T17:52:02.082 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.082 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - handle_peering_op on 3.12 from 1 2022-03-29T17:52:02.082 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,756 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.250366211s))): start 2022-03-29T17:52:02.083 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.250366211s))): got map 15 2022-03-29T17:52:02.083 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.250366211s))): pg present 2022-03-29T17:52:02.083 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.245391846s))): complete 2022-03-29T17:52:02.083 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.250366211s) for pg: 3.12 2022-03-29T17:52:02.084 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.250366211s 2022-03-29T17:52:02.084 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.250366211s (sent 142.250366211s) 2022-03-29T17:52:02.084 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.084 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.250366211s))): submitting ctx 2022-03-29T17:52:02.084 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.085 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:51:59,758 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.1 pg_lease_ack(ruub 142.250366211s))): complete 2022-03-29T17:52:02.085 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:51:59,804 [shard 0] alienstore - stat 2022-03-29T17:52:02.127 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,686 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:02.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:02.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:02.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - peering_event(id=415, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:02.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,686 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.059051514s))): complete 2022-03-29T17:52:02.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,687 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:52:02.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,687 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): start 2022-03-29T17:52:02.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,688 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.062164307s))): pg present 2022-03-29T17:52:02.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.4 2022-03-29T17:52:02.130 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.329620 4 0.004059 2022-03-29T17:52:02.131 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,689 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.3596916, 0.0040599 spent on 4 events 2022-03-29T17:52:02.131 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:02.131 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,689 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:02.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 1,2 upset 1,2 acting_recovery_backfill 1,2 2022-03-29T17:52:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 2 objects 0 missing 0 2022-03-29T17:52:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:02.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:02.134 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000273 0 0.000000 2022-03-29T17:52:02.134 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,689 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.689367, 0.0 spent on 0 events 2022-03-29T17:52:02.134 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:02.134 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,689 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:02.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,689 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:02.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:02.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - peering_event(id=414, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:02.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): got map 15 2022-03-29T17:52:02.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,690 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,692 [shard 0] osd - peering_event(id=415, detail=PeeringEvent(from=1 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:02.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,692 [shard 0] osd - client_request(id=142, detail=m=[osd_op(client.4179.0:3 3.1a 3:5cea7035:::rbd_info:head {read 0~0} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e15) v8]): got obc lock 2022-03-29T17:52:02.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): pg present 2022-03-29T17:52:02.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:02.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.062164307s) for pg: 3.10 2022-03-29T17:52:02.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.062164307s 2022-03-29T17:52:02.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.062164307s (sent 142.062164307s) 2022-03-29T17:52:02.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: peer osd.0 activated and committed 2022-03-29T17:52:02.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:02.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:02.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:02.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,693 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating publish_stats_to_osd 15:4 2022-03-29T17:52:02.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating state: all_activated_and_committed 2022-03-29T17:52:02.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until peer osd.0 ruub 142.062164307s 2022-03-29T17:52:02.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating recalc_readable_until readable_until[_ub] 142.062164307s (sent 142.389938354s) 2022-03-29T17:52:02.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats object_location_counts {} 2022-03-29T17:52:02.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats degraded 0 2022-03-29T17:52:02.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating update_calc_stats misplaced 0 2022-03-29T17:52:02.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active state: mnow 126.390556335s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:02.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - send_pg_created 2022-03-29T17:52:02.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active share_pg_info 2022-03-29T17:52:02.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:5 2022-03-29T17:52:02.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:02.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,694 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_backfill does not need backfill 2022-03-29T17:52:02.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:02.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - peering_event(id=418, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): start 2022-03-29T17:52:02.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:6 2022-03-29T17:52:02.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.062164307s))): submitting ctx 2022-03-29T17:52:02.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - handle_peering_op on 3.15 from 0 2022-03-29T17:52:02.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): start 2022-03-29T17:52:02.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op read 2022-03-29T17:52:02.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - handling op read on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,695 [shard 0] osd - read: 3:5cea7035:::rbd_info:head DNE 2022-03-29T17:52:02.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - rollback_obc_if_modified: object 3:5cea7035:::rbd_info:head got error generic:2, need_rollback=false 2022-03-29T17:52:02.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - peering_event(id=414, detail=PeeringEvent(from=1 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:02.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): got map 15 2022-03-29T17:52:02.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - peering_event(id=418, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): pg present 2022-03-29T17:52:02.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,697 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s) for pg: 3.8 2022-03-29T17:52:02.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s) 2022-03-29T17:52:02.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.394180298s 2022-03-29T17:52:02.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.394180298s 2022-03-29T17:52:02.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:52:02.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,698 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): start 2022-03-29T17:52:02.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,700 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): pg present 2022-03-29T17:52:02.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,700 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,700 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease_ack(ruub 142.062164307s))): complete 2022-03-29T17:52:02.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,700 [shard 0] osd - OpSequencer::finish_op_in_order: op=142, last_started=142, last_unblocked=142, last_completed=141 2022-03-29T17:52:02.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,700 [shard 0] osd - client_request(id=142, detail=m=[osd_op(client.4179.0:3 3.1a 3:5cea7035:::rbd_info:head {read 0~0} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e15) v8]): destroying 2022-03-29T17:52:02.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,700 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): got map 15 2022-03-29T17:52:02.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered for pg: 3.10 2022-03-29T17:52:02.159 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Activating 0.332226 4 0.004927 2022-03-29T17:52:02.159 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,701 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576319.368811, 0.004927383 spent on 4 events 2022-03-29T17:52:02.159 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered 2022-03-29T17:52:02.159 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,701 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:02.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active needs_recovery is recovered 2022-03-29T17:52:02.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats actingset 0,1 upset 0,1 acting_recovery_backfill 0,1 2022-03-29T17:52:02.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:02.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats shard 0 objects 0 missing 0 2022-03-29T17:52:02.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats object_location_counts {} 2022-03-29T17:52:02.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 0 missing= 0 2022-03-29T17:52:02.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:02.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats degraded 0 2022-03-29T17:52:02.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active update_calc_stats misplaced 0 2022-03-29T17:52:02.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active publish_stats_to_osd 15:7 2022-03-29T17:52:02.162 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active/Recovered 0.000312 0 0.000000 2022-03-29T17:52:02.162 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,701 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576319.7011042, 0.0 spent on 0 events 2022-03-29T17:52:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean 2022-03-29T17:52:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,701 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean share_pg_info 2022-03-29T17:52:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:8 2022-03-29T17:52:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - peering_event(id=418, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s) for pg: 3.15 2022-03-29T17:52:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,701 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s) 2022-03-29T17:52:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.397918701s 2022-03-29T17:52:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.397918701s 2022-03-29T17:52:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - handle_peering_op on 3.15 from 0 2022-03-29T17:52:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,702 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): start 2022-03-29T17:52:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,703 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): pg present 2022-03-29T17:52:02.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): complete 2022-03-29T17:52:02.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - peering_event(id=418, detail=PeeringEvent(from=1 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 AllReplicasRecovered)): complete 2022-03-29T17:52:02.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): got map 15 2022-03-29T17:52:02.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): complete 2022-03-29T17:52:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): pg present 2022-03-29T17:52:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s) for pg: 3.8 2022-03-29T17:52:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,704 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s) for pg: 3.15 2022-03-29T17:52:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:52:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): start 2022-03-29T17:52:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,706 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): complete 2022-03-29T17:52:02.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): got map 15 2022-03-29T17:52:02.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): pg present 2022-03-29T17:52:02.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:52:02.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): start 2022-03-29T17:52:02.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s) for pg: 3.8 2022-03-29T17:52:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): complete 2022-03-29T17:52:02.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - handle_peering_op on 3.15 from 0 2022-03-29T17:52:02.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,707 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): start 2022-03-29T17:52:02.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,708 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): got map 15 2022-03-29T17:52:02.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,708 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): got map 15 2022-03-29T17:52:02.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,708 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.8 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.929672241s ub 142.110671997s int 16.000000000s))): complete 2022-03-29T17:52:02.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,708 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): pg present 2022-03-29T17:52:02.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,708 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): pg present 2022-03-29T17:52:02.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,708 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s) for pg: 3.7 2022-03-29T17:52:02.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s) 2022-03-29T17:52:02.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.404953003s 2022-03-29T17:52:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.404953003s 2022-03-29T17:52:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - handle_peering_op on 3.17 from 0 2022-03-29T17:52:02.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): start 2022-03-29T17:52:02.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s) for pg: 3.15 2022-03-29T17:52:02.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.178 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,709 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:52:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): start 2022-03-29T17:52:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): complete 2022-03-29T17:52:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): got map 15 2022-03-29T17:52:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): pg present 2022-03-29T17:52:02.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): got map 15 2022-03-29T17:52:02.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=0 pgid=3.15 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.920486450s ub 142.137985229s int 16.000000000s))): complete 2022-03-29T17:52:02.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - handle_peering_op on 3.17 from 0 2022-03-29T17:52:02.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,710 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): start 2022-03-29T17:52:02.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): pg present 2022-03-29T17:52:02.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:52:02.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): start 2022-03-29T17:52:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s) for pg: 3.17 2022-03-29T17:52:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s) 2022-03-29T17:52:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.407394409s 2022-03-29T17:52:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.407394409s 2022-03-29T17:52:02.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): got map 15 2022-03-29T17:52:02.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): got map 15 2022-03-29T17:52:02.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s) for pg: 3.7 2022-03-29T17:52:02.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,711 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): pg present 2022-03-29T17:52:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): pg present 2022-03-29T17:52:02.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): complete 2022-03-29T17:52:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s) for pg: 3.17 2022-03-29T17:52:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s) for pg: 3.7 2022-03-29T17:52:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,712 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): complete 2022-03-29T17:52:02.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - handle_peering_op on 3.17 from 0 2022-03-29T17:52:02.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): start 2022-03-29T17:52:02.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:52:02.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): start 2022-03-29T17:52:02.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): got map 15 2022-03-29T17:52:02.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): pg present 2022-03-29T17:52:02.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): complete 2022-03-29T17:52:02.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): got map 15 2022-03-29T17:52:02.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - handle_peering_op on 3.6 from 0 2022-03-29T17:52:02.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,713 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): start 2022-03-29T17:52:02.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): pg present 2022-03-29T17:52:02.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:52:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): start 2022-03-29T17:52:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): got map 15 2022-03-29T17:52:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s) for pg: 3.17 2022-03-29T17:52:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): got map 15 2022-03-29T17:52:02.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): pg present 2022-03-29T17:52:02.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s) for pg: 3.1e 2022-03-29T17:52:02.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s) 2022-03-29T17:52:02.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.410812378s 2022-03-29T17:52:02.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.410812378s 2022-03-29T17:52:02.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,714 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.7 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.933349609s ub 142.129089355s int 16.000000000s))): complete 2022-03-29T17:52:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): pg present 2022-03-29T17:52:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=0 pgid=3.17 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.926925659s ub 142.165069580s int 16.000000000s))): complete 2022-03-29T17:52:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s) for pg: 3.1e 2022-03-29T17:52:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s) for pg: 3.6 2022-03-29T17:52:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s) 2022-03-29T17:52:02.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.411605835s 2022-03-29T17:52:02.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.411605835s 2022-03-29T17:52:02.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - handle_peering_op on 3.6 from 0 2022-03-29T17:52:02.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,715 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): start 2022-03-29T17:52:02.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,716 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:52:02.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,716 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): start 2022-03-29T17:52:02.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,716 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): got map 15 2022-03-29T17:52:02.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,716 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): complete 2022-03-29T17:52:02.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,716 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): complete 2022-03-29T17:52:02.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,716 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): pg present 2022-03-29T17:52:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,716 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): got map 15 2022-03-29T17:52:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,716 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): pg present 2022-03-29T17:52:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s) for pg: 3.6 2022-03-29T17:52:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s) for pg: 3.1e 2022-03-29T17:52:02.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): complete 2022-03-29T17:52:02.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:52:02.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): start 2022-03-29T17:52:02.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): complete 2022-03-29T17:52:02.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - handle_peering_op on 3.6 from 0 2022-03-29T17:52:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,717 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): start 2022-03-29T17:52:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): got map 15 2022-03-29T17:52:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.1e sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.943237305s ub 142.150024414s int 16.000000000s))): complete 2022-03-29T17:52:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): got map 15 2022-03-29T17:52:02.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): pg present 2022-03-29T17:52:02.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): pg present 2022-03-29T17:52:02.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s) for pg: 3.16 2022-03-29T17:52:02.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s) 2022-03-29T17:52:02.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.414535522s 2022-03-29T17:52:02.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.414535522s 2022-03-29T17:52:02.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:52:02.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,718 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): start 2022-03-29T17:52:02.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - handle_peering_op on 3.1b from 0 2022-03-29T17:52:02.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): start 2022-03-29T17:52:02.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s) for pg: 3.6 2022-03-29T17:52:02.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): complete 2022-03-29T17:52:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): got map 15 2022-03-29T17:52:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): got map 15 2022-03-29T17:52:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): pg present 2022-03-29T17:52:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,719 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): pg present 2022-03-29T17:52:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - handle_peering_op on 3.1b from 0 2022-03-29T17:52:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): start 2022-03-29T17:52:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:52:02.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): start 2022-03-29T17:52:02.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s) for pg: 3.16 2022-03-29T17:52:02.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=0 pgid=3.6 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.931915283s ub 142.180969238s int 16.000000000s))): complete 2022-03-29T17:52:02.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): got map 15 2022-03-29T17:52:02.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s) for pg: 3.1b 2022-03-29T17:52:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s) 2022-03-29T17:52:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.416732788s 2022-03-29T17:52:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.416732788s 2022-03-29T17:52:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,720 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): got map 15 2022-03-29T17:52:02.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): pg present 2022-03-29T17:52:02.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): pg present 2022-03-29T17:52:02.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): complete 2022-03-29T17:52:02.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s) for pg: 3.1b 2022-03-29T17:52:02.222 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - handle_peering_op on 3.1b from 0 2022-03-29T17:52:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): start 2022-03-29T17:52:02.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s) for pg: 3.16 2022-03-29T17:52:02.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,721 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.224 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): complete 2022-03-29T17:52:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): got map 15 2022-03-29T17:52:02.226 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): pg present 2022-03-29T17:52:02.226 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): complete 2022-03-29T17:52:02.226 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:52:02.226 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): start 2022-03-29T17:52:02.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.16 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.947082520s ub 142.175064087s int 16.000000000s))): complete 2022-03-29T17:52:02.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - handle_peering_op on 3.f from 0 2022-03-29T17:52:02.227 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,722 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): start 2022-03-29T17:52:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s) for pg: 3.1b 2022-03-29T17:52:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): got map 15 2022-03-29T17:52:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): pg present 2022-03-29T17:52:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): got map 15 2022-03-29T17:52:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): pg present 2022-03-29T17:52:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:52:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): start 2022-03-29T17:52:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s) for pg: 3.5 2022-03-29T17:52:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s) 2022-03-29T17:52:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.419830322s 2022-03-29T17:52:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.419830322s 2022-03-29T17:52:02.231 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,723 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - handle_peering_op on 3.f from 0 2022-03-29T17:52:02.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): start 2022-03-29T17:52:02.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=0 pgid=3.1b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.935989380s ub 142.197250366s int 16.000000000s))): complete 2022-03-29T17:52:02.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): got map 15 2022-03-29T17:52:02.233 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): got map 15 2022-03-29T17:52:02.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): pg present 2022-03-29T17:52:02.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s) for pg: 3.f 2022-03-29T17:52:02.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s) 2022-03-29T17:52:02.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.420822144s 2022-03-29T17:52:02.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.420822144s 2022-03-29T17:52:02.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,724 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): pg present 2022-03-29T17:52:02.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): complete 2022-03-29T17:52:02.236 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - handle_peering_op on 3.f from 0 2022-03-29T17:52:02.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): start 2022-03-29T17:52:02.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s) for pg: 3.f 2022-03-29T17:52:02.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.237 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s) for pg: 3.5 2022-03-29T17:52:02.238 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,725 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): complete 2022-03-29T17:52:02.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): got map 15 2022-03-29T17:52:02.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:52:02.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): start 2022-03-29T17:52:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): pg present 2022-03-29T17:52:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): got map 15 2022-03-29T17:52:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): complete 2022-03-29T17:52:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): pg present 2022-03-29T17:52:02.241 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s) for pg: 3.f 2022-03-29T17:52:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,726 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): complete 2022-03-29T17:52:02.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - handle_peering_op on 3.19 from 2 2022-03-29T17:52:02.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.247009277s))): start 2022-03-29T17:52:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s) for pg: 3.5 2022-03-29T17:52:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=0 pgid=3.f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.953430176s ub 142.209915161s int 16.000000000s))): complete 2022-03-29T17:52:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - handle_peering_op on 3.a from 0 2022-03-29T17:52:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): start 2022-03-29T17:52:02.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,727 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.247009277s))): got map 15 2022-03-29T17:52:02.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.247009277s))): pg present 2022-03-29T17:52:02.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.5 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.951522827s ub 142.193099976s int 16.000000000s))): complete 2022-03-29T17:52:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.247009277s) for pg: 3.19 2022-03-29T17:52:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.247009277s 2022-03-29T17:52:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.247009277s (sent 142.247009277s) 2022-03-29T17:52:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.247009277s))): submitting ctx 2022-03-29T17:52:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): got map 15 2022-03-29T17:52:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - handle_peering_op on 3.a from 0 2022-03-29T17:52:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): start 2022-03-29T17:52:02.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): pg present 2022-03-29T17:52:02.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.19 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.247009277s))): complete 2022-03-29T17:52:02.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,728 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): got map 15 2022-03-29T17:52:02.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): pg present 2022-03-29T17:52:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s) for pg: 3.a 2022-03-29T17:52:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s) 2022-03-29T17:52:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.425125122s 2022-03-29T17:52:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.425125122s 2022-03-29T17:52:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - handle_peering_op on 3.a from 0 2022-03-29T17:52:02.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): start 2022-03-29T17:52:02.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s) for pg: 3.a 2022-03-29T17:52:02.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,729 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): complete 2022-03-29T17:52:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:52:02.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): start 2022-03-29T17:52:02.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): got map 15 2022-03-29T17:52:02.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): got map 15 2022-03-29T17:52:02.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): complete 2022-03-29T17:52:02.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): pg present 2022-03-29T17:52:02.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): pg present 2022-03-29T17:52:02.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s) for pg: 3.a 2022-03-29T17:52:02.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,730 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s) for pg: 3.11 2022-03-29T17:52:02.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s) 2022-03-29T17:52:02.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.427017212s 2022-03-29T17:52:02.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.427017212s 2022-03-29T17:52:02.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:52:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): start 2022-03-29T17:52:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - handle_peering_op on 3.c from 0 2022-03-29T17:52:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): start 2022-03-29T17:52:02.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): got map 15 2022-03-29T17:52:02.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,731 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): pg present 2022-03-29T17:52:02.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): got map 15 2022-03-29T17:52:02.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=0 pgid=3.a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.957992554s ub 142.225738525s int 16.000000000s))): complete 2022-03-29T17:52:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.2 pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): complete 2022-03-29T17:52:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): pg present 2022-03-29T17:52:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s) for pg: 3.11 2022-03-29T17:52:02.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:52:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): start 2022-03-29T17:52:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s) for pg: 3.c 2022-03-29T17:52:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s) 2022-03-29T17:52:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.428665161s 2022-03-29T17:52:02.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.428665161s 2022-03-29T17:52:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,732 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): got map 15 2022-03-29T17:52:02.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - handle_peering_op on 3.c from 0 2022-03-29T17:52:02.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): start 2022-03-29T17:52:02.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): pg present 2022-03-29T17:52:02.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): complete 2022-03-29T17:52:02.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): got map 15 2022-03-29T17:52:02.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): pg present 2022-03-29T17:52:02.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s) for pg: 3.11 2022-03-29T17:52:02.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,733 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - handle_peering_op on 3.14 from 2 2022-03-29T17:52:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.294967651s))): start 2022-03-29T17:52:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): complete 2022-03-29T17:52:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s) for pg: 3.c 2022-03-29T17:52:02.268 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - handle_peering_op on 3.c from 0 2022-03-29T17:52:02.269 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): start 2022-03-29T17:52:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,734 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.294967651s))): got map 15 2022-03-29T17:52:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.11 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 2 info: 3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.956466675s ub 142.221298218s int 16.000000000s))): complete 2022-03-29T17:52:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.294967651s))): pg present 2022-03-29T17:52:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): got map 15 2022-03-29T17:52:02.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): pg present 2022-03-29T17:52:02.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - handle_peering_op on 3.1f from 0 2022-03-29T17:52:02.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): start 2022-03-29T17:52:02.271 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.294967651s) for pg: 3.14 2022-03-29T17:52:02.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.294967651s 2022-03-29T17:52:02.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.294967651s (sent 142.294967651s) 2022-03-29T17:52:02.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.294967651s))): submitting ctx 2022-03-29T17:52:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,735 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): complete 2022-03-29T17:52:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): got map 15 2022-03-29T17:52:02.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s) for pg: 3.c 2022-03-29T17:52:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): pg present 2022-03-29T17:52:02.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.14 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.294967651s))): complete 2022-03-29T17:52:02.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - handle_peering_op on 3.1f from 0 2022-03-29T17:52:02.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): start 2022-03-29T17:52:02.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s) for pg: 3.1f 2022-03-29T17:52:02.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s) 2022-03-29T17:52:02.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.432769775s 2022-03-29T17:52:02.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.432769775s 2022-03-29T17:52:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,736 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=0 pgid=3.c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.962356567s ub 142.233489990s int 16.000000000s))): complete 2022-03-29T17:52:02.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): got map 15 2022-03-29T17:52:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): pg present 2022-03-29T17:52:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s) for pg: 3.1f 2022-03-29T17:52:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - handle_peering_op on 3.12 from 0 2022-03-29T17:52:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): start 2022-03-29T17:52:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): complete 2022-03-29T17:52:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,737 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): got map 15 2022-03-29T17:52:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): pg present 2022-03-29T17:52:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - handle_peering_op on 3.0 from 2 2022-03-29T17:52:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.317977905s))): start 2022-03-29T17:52:02.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - handle_peering_op on 3.12 from 0 2022-03-29T17:52:02.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): start 2022-03-29T17:52:02.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s) for pg: 3.12 2022-03-29T17:52:02.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s) 2022-03-29T17:52:02.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 142.434600830s 2022-03-29T17:52:02.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 142.434600830s 2022-03-29T17:52:02.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.317977905s))): got map 15 2022-03-29T17:52:02.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,738 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): complete 2022-03-29T17:52:02.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): got map 15 2022-03-29T17:52:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): pg present 2022-03-29T17:52:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.317977905s))): pg present 2022-03-29T17:52:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.317977905s) for pg: 3.0 2022-03-29T17:52:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.317977905s 2022-03-29T17:52:02.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.317977905s (sent 142.317977905s) 2022-03-29T17:52:02.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.317977905s))): submitting ctx 2022-03-29T17:52:02.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLease epoch 15 from osd.0 pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): complete 2022-03-29T17:52:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - handle_peering_op on 3.1f from 0 2022-03-29T17:52:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): start 2022-03-29T17:52:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s) for pg: 3.12 2022-03-29T17:52:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14 2022-03-29T17:52:02.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,739 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,740 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,740 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,745 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): got map 15 2022-03-29T17:52:02.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,745 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.0 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.317977905s))): complete 2022-03-29T17:52:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,745 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): pg present 2022-03-29T17:52:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,745 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): complete 2022-03-29T17:52:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s) for pg: 3.1f 2022-03-29T17:52:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - handle_peering_op on 3.1a from 2 2022-03-29T17:52:02.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.332717896s))): start 2022-03-29T17:52:02.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,746 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.332717896s))): got map 15 2022-03-29T17:52:02.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - handle_peering_op on 3.12 from 0 2022-03-29T17:52:02.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): start 2022-03-29T17:52:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=0 pgid=3.1f sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.966506958s ub 142.245391846s int 16.000000000s))): complete 2022-03-29T17:52:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.332717896s))): pg present 2022-03-29T17:52:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): got map 15 2022-03-29T17:52:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): pg present 2022-03-29T17:52:02.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - handle_peering_op on 3.1c from 0 2022-03-29T17:52:02.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.314071655s))): start 2022-03-29T17:52:02.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.332717896s) for pg: 3.1a 2022-03-29T17:52:02.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.332717896s 2022-03-29T17:52:02.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,747 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.332717896s (sent 142.332717896s) 2022-03-29T17:52:02.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.332717896s))): submitting ctx 2022-03-29T17:52:02.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.314071655s))): got map 15 2022-03-29T17:52:02.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s) for pg: 3.12 2022-03-29T17:52:02.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history advanced history from ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14 2022-03-29T17:52:02.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active update_history clearing past_intervals 2022-03-29T17:52:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): submitting ctx 2022-03-29T17:52:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.314071655s))): pg present 2022-03-29T17:52:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.1a sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.332717896s))): complete 2022-03-29T17:52:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,748 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.314071655s) for pg: 3.1c 2022-03-29T17:52:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.314071655s 2022-03-29T17:52:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.314071655s (sent 142.314071655s) 2022-03-29T17:52:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.314071655s))): submitting ctx 2022-03-29T17:52:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=0 pgid=3.12 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MInfoRec from 0 info: 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) pg_lease(ru 141.971359253s ub 142.250366211s int 16.000000000s))): complete 2022-03-29T17:52:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - handle_peering_op on 3.2 from 2 2022-03-29T17:52:02.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.353500366s))): start 2022-03-29T17:52:02.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=0 pgid=3.1c sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.314071655s))): complete 2022-03-29T17:52:02.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.353500366s))): got map 15 2022-03-29T17:52:02.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,749 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.353500366s))): pg present 2022-03-29T17:52:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - handle_peering_op on 3.b from 0 2022-03-29T17:52:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,757 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.336990356s))): start 2022-03-29T17:52:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,758 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.353500366s) for pg: 3.2 2022-03-29T17:52:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,758 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.353500366s 2022-03-29T17:52:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,758 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.353500366s (sent 142.353500366s) 2022-03-29T17:52:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,758 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,758 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.353500366s))): submitting ctx 2022-03-29T17:52:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,758 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,758 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.336990356s))): got map 15 2022-03-29T17:52:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.336990356s))): pg present 2022-03-29T17:52:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.2 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.353500366s))): complete 2022-03-29T17:52:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.336990356s) for pg: 3.b 2022-03-29T17:52:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.336990356s 2022-03-29T17:52:02.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.336990356s (sent 142.336990356s) 2022-03-29T17:52:02.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.336990356s))): submitting ctx 2022-03-29T17:52:02.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - handle_peering_op on 3.4 from 2 2022-03-29T17:52:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,759 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.370239258s))): start 2022-03-29T17:52:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,760 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=0 pgid=3.b sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.336990356s))): complete 2022-03-29T17:52:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,760 [shard 0] osd - handle_peering_op on 3.13 from 0 2022-03-29T17:52:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,760 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.347534180s))): start 2022-03-29T17:52:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,760 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.370239258s))): got map 15 2022-03-29T17:52:02.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,760 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.370239258s))): pg present 2022-03-29T17:52:02.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,760 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.347534180s))): got map 15 2022-03-29T17:52:02.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,760 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.347534180s))): pg present 2022-03-29T17:52:02.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.370239258s) for pg: 3.4 2022-03-29T17:52:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.370239258s 2022-03-29T17:52:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.370239258s (sent 142.370239258s) 2022-03-29T17:52:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.370239258s))): submitting ctx 2022-03-29T17:52:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.347534180s) for pg: 3.13 2022-03-29T17:52:02.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.347534180s 2022-03-29T17:52:02.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.347534180s (sent 142.347534180s) 2022-03-29T17:52:02.306 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.347534180s))): submitting ctx 2022-03-29T17:52:02.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.4 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.2 pg_lease_ack(ruub 142.370239258s))): complete 2022-03-29T17:52:02.307 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,761 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=0 pgid=3.13 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.347534180s))): complete 2022-03-29T17:52:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - handle_peering_op on 3.d from 0 2022-03-29T17:52:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.373336792s))): start 2022-03-29T17:52:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.373336792s))): got map 15 2022-03-29T17:52:02.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.373336792s))): pg present 2022-03-29T17:52:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.373336792s) for pg: 3.d 2022-03-29T17:52:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.373336792s 2022-03-29T17:52:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.373336792s (sent 142.373336792s) 2022-03-29T17:52:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.373336792s))): submitting ctx 2022-03-29T17:52:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,762 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - handle_peering_op on 3.10 from 0 2022-03-29T17:52:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.389938354s))): start 2022-03-29T17:52:02.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=0 pgid=3.d sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.373336792s))): complete 2022-03-29T17:52:02.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.389938354s))): got map 15 2022-03-29T17:52:02.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.389938354s))): pg present 2022-03-29T17:52:02.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - do_peering_event handling epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.389938354s) for pg: 3.10 2022-03-29T17:52:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.389938354s 2022-03-29T17:52:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.389938354s (sent 142.389938354s) 2022-03-29T17:52:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.389938354s))): submitting ctx 2022-03-29T17:52:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,763 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:51:59,764 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=0 pgid=3.10 sent=15 requested=15 evt=epoch_sent: 15 epoch_requested: 15 MLeaseAck epoch 15 from osd.0 pg_lease_ack(ruub 142.389938354s))): complete 2022-03-29T17:52:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:51:59,818 [shard 0] alienstore - stat 2022-03-29T17:52:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,295 [shard 0] osd - handle_osd_map osd_map(15..16 src has 1..16) v4 2022-03-29T17:52:02.313 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,295 [shard 0] osd - handle_osd_map epochs [15..16], i have 15, src has [1..16] 2022-03-29T17:52:02.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,296 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:02.314 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,297 [shard 0] osd - osd.1: committed_osd_maps(16, 16) 2022-03-29T17:52:02.314 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,297 [shard 0] osd - osd.1: map e 16 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:02.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - client_request(id=143, detail=m=[osd_op(client.4179.0:5 3.1a 3.ac0e573a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e16) v8]): start 2022-03-29T17:52:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - client_request(id=143, detail=m=[osd_op(client.4179.0:5 3.1a 3.ac0e573a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e16) v8]): in repeat 2022-03-29T17:52:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=83, detail=PGAdvanceMap(pg=1.0 from=15 to=16)): start 2022-03-29T17:52:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=84, detail=PGAdvanceMap(pg=2.0 from=15 to=16)): start 2022-03-29T17:52:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=85, detail=PGAdvanceMap(pg=2.1 from=15 to=16)): start 2022-03-29T17:52:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=86, detail=PGAdvanceMap(pg=2.2 from=15 to=16)): start 2022-03-29T17:52:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=87, detail=PGAdvanceMap(pg=2.3 from=15 to=16)): start 2022-03-29T17:52:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=88, detail=PGAdvanceMap(pg=2.4 from=15 to=16)): start 2022-03-29T17:52:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=89, detail=PGAdvanceMap(pg=2.5 from=15 to=16)): start 2022-03-29T17:52:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=90, detail=PGAdvanceMap(pg=2.6 from=15 to=16)): start 2022-03-29T17:52:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=91, detail=PGAdvanceMap(pg=2.7 from=15 to=16)): start 2022-03-29T17:52:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=92, detail=PGAdvanceMap(pg=3.0 from=15 to=16)): start 2022-03-29T17:52:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=93, detail=PGAdvanceMap(pg=3.2 from=15 to=16)): start 2022-03-29T17:52:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=94, detail=PGAdvanceMap(pg=3.4 from=15 to=16)): start 2022-03-29T17:52:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=95, detail=PGAdvanceMap(pg=3.5 from=15 to=16)): start 2022-03-29T17:52:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=96, detail=PGAdvanceMap(pg=3.6 from=15 to=16)): start 2022-03-29T17:52:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=97, detail=PGAdvanceMap(pg=3.7 from=15 to=16)): start 2022-03-29T17:52:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=98, detail=PGAdvanceMap(pg=3.8 from=15 to=16)): start 2022-03-29T17:52:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=99, detail=PGAdvanceMap(pg=3.a from=15 to=16)): start 2022-03-29T17:52:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=100, detail=PGAdvanceMap(pg=3.b from=15 to=16)): start 2022-03-29T17:52:02.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=101, detail=PGAdvanceMap(pg=3.c from=15 to=16)): start 2022-03-29T17:52:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=102, detail=PGAdvanceMap(pg=3.d from=15 to=16)): start 2022-03-29T17:52:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=103, detail=PGAdvanceMap(pg=3.f from=15 to=16)): start 2022-03-29T17:52:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=104, detail=PGAdvanceMap(pg=3.10 from=15 to=16)): start 2022-03-29T17:52:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=105, detail=PGAdvanceMap(pg=3.11 from=15 to=16)): start 2022-03-29T17:52:02.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=106, detail=PGAdvanceMap(pg=3.12 from=15 to=16)): start 2022-03-29T17:52:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=107, detail=PGAdvanceMap(pg=3.13 from=15 to=16)): start 2022-03-29T17:52:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=108, detail=PGAdvanceMap(pg=3.14 from=15 to=16)): start 2022-03-29T17:52:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=109, detail=PGAdvanceMap(pg=3.15 from=15 to=16)): start 2022-03-29T17:52:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=110, detail=PGAdvanceMap(pg=3.16 from=15 to=16)): start 2022-03-29T17:52:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=111, detail=PGAdvanceMap(pg=3.17 from=15 to=16)): start 2022-03-29T17:52:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=112, detail=PGAdvanceMap(pg=3.19 from=15 to=16)): start 2022-03-29T17:52:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=113, detail=PGAdvanceMap(pg=3.1a from=15 to=16)): start 2022-03-29T17:52:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=114, detail=PGAdvanceMap(pg=3.1b from=15 to=16)): start 2022-03-29T17:52:02.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=115, detail=PGAdvanceMap(pg=3.1c from=15 to=16)): start 2022-03-29T17:52:02.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=116, detail=PGAdvanceMap(pg=3.1e from=15 to=16)): start 2022-03-29T17:52:02.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,298 [shard 0] osd - pg_advance_map(id=117, detail=PGAdvanceMap(pg=3.1f from=15 to=16)): start 2022-03-29T17:52:02.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.323 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,299 [shard 0] osd - evt epoch is 16, i have 15, will wait 2022-03-29T17:52:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,299 [shard 0] osd - osdmap_subscribe(15) 2022-03-29T17:52:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active advmap 2022-03-29T17:52:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Started advmap 2022-03-29T17:52:02.326 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,299 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.334 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.335 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,300 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,301 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,302 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.357 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:12 2022-03-29T17:52:02.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.358 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.364 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,303 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.366 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.367 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.368 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:12 2022-03-29T17:52:02.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:12 2022-03-29T17:52:02.373 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map 2022-03-29T17:52:02.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: handle ActMap primary 2022-03-29T17:52:02.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,304 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean publish_stats_to_osd 15:15 2022-03-29T17:52:02.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.380 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.381 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.388 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,305 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.389 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.391 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:12 2022-03-29T17:52:02.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.395 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.396 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.397 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.398 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,306 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.399 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,316 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.402 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:12 2022-03-29T17:52:02.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 16 2022-03-29T17:52:02.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.408 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.412 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.417 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.418 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,318 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.422 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,319 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.423 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=87, detail=PGAdvanceMap(pg=2.3 from=15 to=16)): complete 2022-03-29T17:52:02.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=86, detail=PGAdvanceMap(pg=2.2 from=15 to=16)): complete 2022-03-29T17:52:02.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=117, detail=PGAdvanceMap(pg=3.1f from=15 to=16)): complete 2022-03-29T17:52:02.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=113, detail=PGAdvanceMap(pg=3.1a from=15 to=16)): complete 2022-03-29T17:52:02.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=85, detail=PGAdvanceMap(pg=2.1 from=15 to=16)): complete 2022-03-29T17:52:02.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=92, detail=PGAdvanceMap(pg=3.0 from=15 to=16)): complete 2022-03-29T17:52:02.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=88, detail=PGAdvanceMap(pg=2.4 from=15 to=16)): complete 2022-03-29T17:52:02.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=107, detail=PGAdvanceMap(pg=3.13 from=15 to=16)): complete 2022-03-29T17:52:02.428 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,320 [shard 0] osd - pg_advance_map(id=116, detail=PGAdvanceMap(pg=3.1e from=15 to=16)): complete 2022-03-29T17:52:02.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - pg_advance_map(id=96, detail=PGAdvanceMap(pg=3.6 from=15 to=16)): complete 2022-03-29T17:52:02.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 16 2022-03-29T17:52:02.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.429 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - pg_advance_map(id=99, detail=PGAdvanceMap(pg=3.a from=15 to=16)): complete 2022-03-29T17:52:02.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - pg_advance_map(id=115, detail=PGAdvanceMap(pg=3.1c from=15 to=16)): complete 2022-03-29T17:52:02.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - pg_advance_map(id=91, detail=PGAdvanceMap(pg=2.7 from=15 to=16)): complete 2022-03-29T17:52:02.430 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,321 [shard 0] osd - pg_advance_map(id=90, detail=PGAdvanceMap(pg=2.6 from=15 to=16)): complete 2022-03-29T17:52:02.431 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,321 [shard 0] osd - handle_osd_map osd_map(16..16 src has 1..16) v4 2022-03-29T17:52:02.431 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,321 [shard 0] osd - handle_osd_map epochs [16..16], i have 16, src has [1..16] 2022-03-29T17:52:02.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=83, detail=PGAdvanceMap(pg=1.0 from=15 to=16)): complete 2022-03-29T17:52:02.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=84, detail=PGAdvanceMap(pg=2.0 from=15 to=16)): complete 2022-03-29T17:52:02.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=111, detail=PGAdvanceMap(pg=3.17 from=15 to=16)): complete 2022-03-29T17:52:02.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=114, detail=PGAdvanceMap(pg=3.1b from=15 to=16)): complete 2022-03-29T17:52:02.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=109, detail=PGAdvanceMap(pg=3.15 from=15 to=16)): complete 2022-03-29T17:52:02.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=108, detail=PGAdvanceMap(pg=3.14 from=15 to=16)): complete 2022-03-29T17:52:02.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=106, detail=PGAdvanceMap(pg=3.12 from=15 to=16)): complete 2022-03-29T17:52:02.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=100, detail=PGAdvanceMap(pg=3.b from=15 to=16)): complete 2022-03-29T17:52:02.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=101, detail=PGAdvanceMap(pg=3.c from=15 to=16)): complete 2022-03-29T17:52:02.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,322 [shard 0] osd - pg_advance_map(id=89, detail=PGAdvanceMap(pg=2.5 from=15 to=16)): complete 2022-03-29T17:52:02.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=105, detail=PGAdvanceMap(pg=3.11 from=15 to=16)): complete 2022-03-29T17:52:02.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=104, detail=PGAdvanceMap(pg=3.10 from=15 to=16)): complete 2022-03-29T17:52:02.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=94, detail=PGAdvanceMap(pg=3.4 from=15 to=16)): complete 2022-03-29T17:52:02.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=93, detail=PGAdvanceMap(pg=3.2 from=15 to=16)): complete 2022-03-29T17:52:02.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=102, detail=PGAdvanceMap(pg=3.d from=15 to=16)): complete 2022-03-29T17:52:02.434 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=110, detail=PGAdvanceMap(pg=3.16 from=15 to=16)): complete 2022-03-29T17:52:02.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=97, detail=PGAdvanceMap(pg=3.7 from=15 to=16)): complete 2022-03-29T17:52:02.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=103, detail=PGAdvanceMap(pg=3.f from=15 to=16)): complete 2022-03-29T17:52:02.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=112, detail=PGAdvanceMap(pg=3.19 from=15 to=16)): complete 2022-03-29T17:52:02.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,324 [shard 0] osd - pg_advance_map(id=98, detail=PGAdvanceMap(pg=3.8 from=15 to=16)): complete 2022-03-29T17:52:02.435 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - pg_advance_map(id=95, detail=PGAdvanceMap(pg=3.5 from=15 to=16)): complete 2022-03-29T17:52:02.436 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,325 [shard 0] osd - osd.1: now active 2022-03-29T17:52:02.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - client_request(id=143, detail=m=[osd_op(client.4179.0:5 3.1a 3.ac0e573a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e16) v8]) same_interval_since: 14 2022-03-29T17:52:02.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - OpSequencer::start_op: op=143, last_started=142, last_unblocked=142, last_completed=142 2022-03-29T17:52:02.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.436 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - client_request(id=143, detail=m=[osd_op(client.4179.0:5 3.1a 3:5cea7035:::rbd_info:head {create, write 0~8 in=8b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e16) v8]): got obc lock 2022-03-29T17:52:02.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:02.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op create 2022-03-29T17:52:02.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - handling op create on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.437 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op write 2022-03-29T17:52:02.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - handling op write on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:02.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head submitting txn 2022-03-29T17:52:02.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:02.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - pg_epoch 16 pg[3.1a( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {16'1 (0'0) modify 3:5cea7035:::rbd_info:head by client.4179.0:5 2022-03-29T17:52:00.297709+0000 0 ObjectCleanRegions clean_offsets: [8~18446744073709551607], clean_omap: 1, new_object: 0} 2022-03-29T17:52:02.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - pg_epoch 16 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 16'1 (0'0) modify 3:5cea7035:::rbd_info:head by client.4179.0:5 2022-03-29T17:52:00.297709+0000 0 ObjectCleanRegions clean_offsets: [8~18446744073709551607], clean_omap: 1, new_object: 0 2022-03-29T17:52:02.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - pg_epoch 16 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 luod=0'0 lua=0'0 crt=16'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:02.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - pg_epoch 16 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 luod=0'0 lua=0'0 crt=16'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:02.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,325 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,326 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 16'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,326 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:02.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:00,327 [shard 0] osd - replicated_request(id=1, detail=RepRequest(from=1 req=osd_repop(client.4179.0:5 3.1a e16/14 3:5cea7035:::rbd_info:head v 16'1, mlcod=16'1) v3)) start 2022-03-29T17:52:02.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:00,327 [shard 0] osd - pg_epoch 15 pg[3.1a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {16'1 (0'0) modify 3:5cea7035:::rbd_info:head by client.4179.0:5 2022-03-29T17:52:00.297709+0000 0 ObjectCleanRegions clean_offsets: [8~18446744073709551607], clean_omap: 1, new_object: 0} 2022-03-29T17:52:02.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:00,327 [shard 0] osd - pg_epoch 15 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 16'1 (0'0) modify 3:5cea7035:::rbd_info:head by client.4179.0:5 2022-03-29T17:52:00.297709+0000 0 ObjectCleanRegions clean_offsets: [8~18446744073709551607], clean_omap: 1, new_object: 0 2022-03-29T17:52:02.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:00,327 [shard 0] osd - pg_epoch 15 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:02.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:00,327 [shard 0] osd - pg_epoch 15 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:02.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:00,327 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:00,327 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 16'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:00,327 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:02.444 INFO:tasks.ceph.osd.2.smithi138.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.1.0-56-g60fdd357/rpm/el8/BUILD/ceph-17.1.0-56-g60fdd357/src/include/inline_memory.h:112:9: runtime error: load of misaligned address 0x61d00019508e for type 'uint32_t', which requires 4 byte alignment 2022-03-29T17:52:02.444 INFO:tasks.ceph.osd.2.smithi138.stderr:0x61d00019508e: note: pointer points here 2022-03-29T17:52:02.444 INFO:tasks.ceph.osd.2.smithi138.stderr: 08 00 00 00 76 61 6c 69 64 61 74 65 03 00 00 00 1f 00 00 00 30 30 30 30 30 30 30 30 31 36 2e 30 2022-03-29T17:52:02.444 INFO:tasks.ceph.osd.2.smithi138.stderr: ^ 2022-03-29T17:52:02.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,368 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:5 3.1a 3:5cea7035:::rbd_info:head {create, write 0~8 in=8b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e16) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:02.446 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,369 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.448 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,369 [shard 0] osd - OpSequencer::finish_op_in_order: op=143, last_started=143, last_unblocked=143, last_completed=142 2022-03-29T17:52:02.448 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:00,369 [shard 0] osd - client_request(id=143, detail=m=[osd_op(client.4179.0:5 3.1a 3:5cea7035:::rbd_info:head {create, write 0~8 in=8b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e16) v8]): destroying 2022-03-29T17:52:02.450 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:00,811 [shard 0] alienstore - stat 2022-03-29T17:52:02.450 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:00,819 [shard 0] alienstore - stat 2022-03-29T17:52:02.493 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:00,827 [shard 0] alienstore - stat 2022-03-29T17:52:02.494 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:01,301 [shard 0] osd - handle_osd_map osd_map(16..17 src has 1..17) v4 2022-03-29T17:52:02.494 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:01,301 [shard 0] osd - handle_osd_map epochs [16..17], i have 15, src has [1..17] 2022-03-29T17:52:02.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,303 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:02.495 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:01,304 [shard 0] osd - osd.2: committed_osd_maps(16, 17) 2022-03-29T17:52:02.495 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:01,304 [shard 0] osd - osd.2: map e 17 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:02.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=42, detail=PGAdvanceMap(pg=2.0 from=15 to=17)): start 2022-03-29T17:52:02.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=43, detail=PGAdvanceMap(pg=2.1 from=15 to=17)): start 2022-03-29T17:52:02.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=44, detail=PGAdvanceMap(pg=2.3 from=15 to=17)): start 2022-03-29T17:52:02.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=45, detail=PGAdvanceMap(pg=3.0 from=15 to=17)): start 2022-03-29T17:52:02.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=46, detail=PGAdvanceMap(pg=3.1 from=15 to=17)): start 2022-03-29T17:52:02.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=47, detail=PGAdvanceMap(pg=3.2 from=15 to=17)): start 2022-03-29T17:52:02.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=48, detail=PGAdvanceMap(pg=3.3 from=15 to=17)): start 2022-03-29T17:52:02.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=49, detail=PGAdvanceMap(pg=3.4 from=15 to=17)): start 2022-03-29T17:52:02.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=50, detail=PGAdvanceMap(pg=3.5 from=15 to=17)): start 2022-03-29T17:52:02.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=51, detail=PGAdvanceMap(pg=3.7 from=15 to=17)): start 2022-03-29T17:52:02.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=52, detail=PGAdvanceMap(pg=3.8 from=15 to=17)): start 2022-03-29T17:52:02.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=53, detail=PGAdvanceMap(pg=3.9 from=15 to=17)): start 2022-03-29T17:52:02.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=54, detail=PGAdvanceMap(pg=3.e from=15 to=17)): start 2022-03-29T17:52:02.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=55, detail=PGAdvanceMap(pg=3.11 from=15 to=17)): start 2022-03-29T17:52:02.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=56, detail=PGAdvanceMap(pg=3.14 from=15 to=17)): start 2022-03-29T17:52:02.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=57, detail=PGAdvanceMap(pg=3.16 from=15 to=17)): start 2022-03-29T17:52:02.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=58, detail=PGAdvanceMap(pg=3.18 from=15 to=17)): start 2022-03-29T17:52:02.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=59, detail=PGAdvanceMap(pg=3.19 from=15 to=17)): start 2022-03-29T17:52:02.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=60, detail=PGAdvanceMap(pg=3.1a from=15 to=17)): start 2022-03-29T17:52:02.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=61, detail=PGAdvanceMap(pg=3.1d from=15 to=17)): start 2022-03-29T17:52:02.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_advance_map(id=62, detail=PGAdvanceMap(pg=3.1e from=15 to=17)): start 2022-03-29T17:52:02.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_epoch 15 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,305 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 15 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 15 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 15 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 16 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,306 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 15 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 15 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 15 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,307 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 15 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 15 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 15 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 16 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,308 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 16 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 16 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 16 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,309 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 15 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 15 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,310 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 16 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,311 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:12 2022-03-29T17:52:02.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 16 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 16 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 15 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 16 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active state: Started advmap 2022-03-29T17:52:02.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 16 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 15 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,312 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,313 [shard 0] osd - client_request(id=144, detail=m=[osd_op(client.4179.0:7 3.1a 3.ac0e573a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - client_request(id=144, detail=m=[osd_op(client.4179.0:7 3.1a 3.ac0e573a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.552 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,314 [shard 0] osd - evt epoch is 17, i have 16, will wait 2022-03-29T17:52:02.552 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,314 [shard 0] osd - osdmap_subscribe(16) 2022-03-29T17:52:02.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:12 2022-03-29T17:52:02.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - pg_epoch 16 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,314 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.556 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,315 [shard 0] osd - handle_osd_map osd_map(17..17 src has 1..17) v4 2022-03-29T17:52:02.556 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,315 [shard 0] osd - handle_osd_map epochs [17..17], i have 16, src has [1..17] 2022-03-29T17:52:02.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 16 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 16 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 16 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,315 [shard 0] osd - pg_epoch 16 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - pg_epoch 16 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active state: Started advmap 2022-03-29T17:52:02.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,316 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,317 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:02.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,317 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,317 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,317 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,317 [shard 0] osd - pg_epoch 15 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,317 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.576 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,318 [shard 0] osd - osd.1: committed_osd_maps(17, 17) 2022-03-29T17:52:02.576 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,318 [shard 0] osd - osd.1: map e 17 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:02.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,318 [shard 0] osd - pg_advance_map(id=118, detail=PGAdvanceMap(pg=1.0 from=16 to=17)): start 2022-03-29T17:52:02.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=119, detail=PGAdvanceMap(pg=2.0 from=16 to=17)): start 2022-03-29T17:52:02.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=120, detail=PGAdvanceMap(pg=2.1 from=16 to=17)): start 2022-03-29T17:52:02.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=121, detail=PGAdvanceMap(pg=2.2 from=16 to=17)): start 2022-03-29T17:52:02.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=122, detail=PGAdvanceMap(pg=2.3 from=16 to=17)): start 2022-03-29T17:52:02.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=123, detail=PGAdvanceMap(pg=2.4 from=16 to=17)): start 2022-03-29T17:52:02.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=124, detail=PGAdvanceMap(pg=2.5 from=16 to=17)): start 2022-03-29T17:52:02.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=125, detail=PGAdvanceMap(pg=2.6 from=16 to=17)): start 2022-03-29T17:52:02.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=126, detail=PGAdvanceMap(pg=2.7 from=16 to=17)): start 2022-03-29T17:52:02.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=127, detail=PGAdvanceMap(pg=3.0 from=16 to=17)): start 2022-03-29T17:52:02.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=128, detail=PGAdvanceMap(pg=3.2 from=16 to=17)): start 2022-03-29T17:52:02.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=129, detail=PGAdvanceMap(pg=3.4 from=16 to=17)): start 2022-03-29T17:52:02.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=130, detail=PGAdvanceMap(pg=3.5 from=16 to=17)): start 2022-03-29T17:52:02.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=131, detail=PGAdvanceMap(pg=3.6 from=16 to=17)): start 2022-03-29T17:52:02.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=132, detail=PGAdvanceMap(pg=3.7 from=16 to=17)): start 2022-03-29T17:52:02.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=133, detail=PGAdvanceMap(pg=3.8 from=16 to=17)): start 2022-03-29T17:52:02.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=134, detail=PGAdvanceMap(pg=3.a from=16 to=17)): start 2022-03-29T17:52:02.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=135, detail=PGAdvanceMap(pg=3.b from=16 to=17)): start 2022-03-29T17:52:02.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=136, detail=PGAdvanceMap(pg=3.c from=16 to=17)): start 2022-03-29T17:52:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=137, detail=PGAdvanceMap(pg=3.d from=16 to=17)): start 2022-03-29T17:52:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=138, detail=PGAdvanceMap(pg=3.f from=16 to=17)): start 2022-03-29T17:52:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=139, detail=PGAdvanceMap(pg=3.10 from=16 to=17)): start 2022-03-29T17:52:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=140, detail=PGAdvanceMap(pg=3.11 from=16 to=17)): start 2022-03-29T17:52:02.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=141, detail=PGAdvanceMap(pg=3.12 from=16 to=17)): start 2022-03-29T17:52:02.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=142, detail=PGAdvanceMap(pg=3.13 from=16 to=17)): start 2022-03-29T17:52:02.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_advance_map(id=143, detail=PGAdvanceMap(pg=3.14 from=16 to=17)): start 2022-03-29T17:52:02.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,319 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=144, detail=PGAdvanceMap(pg=3.15 from=16 to=17)): start 2022-03-29T17:52:02.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=145, detail=PGAdvanceMap(pg=3.16 from=16 to=17)): start 2022-03-29T17:52:02.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=146, detail=PGAdvanceMap(pg=3.17 from=16 to=17)): start 2022-03-29T17:52:02.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=147, detail=PGAdvanceMap(pg=3.19 from=16 to=17)): start 2022-03-29T17:52:02.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=148, detail=PGAdvanceMap(pg=3.1a from=16 to=17)): start 2022-03-29T17:52:02.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=149, detail=PGAdvanceMap(pg=3.1b from=16 to=17)): start 2022-03-29T17:52:02.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=150, detail=PGAdvanceMap(pg=3.1c from=16 to=17)): start 2022-03-29T17:52:02.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=151, detail=PGAdvanceMap(pg=3.1e from=16 to=17)): start 2022-03-29T17:52:02.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=152, detail=PGAdvanceMap(pg=3.1f from=16 to=17)): start 2022-03-29T17:52:02.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_advance_map(id=49, detail=PGAdvanceMap(pg=3.4 from=15 to=17)): complete 2022-03-29T17:52:02.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active activate_map 2022-03-29T17:52:02.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_epoch 16 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_epoch 16 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_advance_map(id=52, detail=PGAdvanceMap(pg=3.8 from=15 to=17)): complete 2022-03-29T17:52:02.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_advance_map(id=55, detail=PGAdvanceMap(pg=3.11 from=15 to=17)): complete 2022-03-29T17:52:02.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_advance_map(id=46, detail=PGAdvanceMap(pg=3.1 from=15 to=17)): complete 2022-03-29T17:52:02.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_advance_map(id=58, detail=PGAdvanceMap(pg=3.18 from=15 to=17)): complete 2022-03-29T17:52:02.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,321 [shard 0] osd - pg_advance_map(id=48, detail=PGAdvanceMap(pg=3.3 from=15 to=17)): complete 2022-03-29T17:52:02.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=50, detail=PGAdvanceMap(pg=3.5 from=15 to=17)): complete 2022-03-29T17:52:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 16 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 16 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=57, detail=PGAdvanceMap(pg=3.16 from=15 to=17)): complete 2022-03-29T17:52:02.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=51, detail=PGAdvanceMap(pg=3.7 from=15 to=17)): complete 2022-03-29T17:52:02.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=54, detail=PGAdvanceMap(pg=3.e from=15 to=17)): complete 2022-03-29T17:52:02.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=59, detail=PGAdvanceMap(pg=3.19 from=15 to=17)): complete 2022-03-29T17:52:02.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=42, detail=PGAdvanceMap(pg=2.0 from=15 to=17)): complete 2022-03-29T17:52:02.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=53, detail=PGAdvanceMap(pg=3.9 from=15 to=17)): complete 2022-03-29T17:52:02.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=44, detail=PGAdvanceMap(pg=2.3 from=15 to=17)): complete 2022-03-29T17:52:02.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=61, detail=PGAdvanceMap(pg=3.1d from=15 to=17)): complete 2022-03-29T17:52:02.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=43, detail=PGAdvanceMap(pg=2.1 from=15 to=17)): complete 2022-03-29T17:52:02.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=56, detail=PGAdvanceMap(pg=3.14 from=15 to=17)): complete 2022-03-29T17:52:02.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=45, detail=PGAdvanceMap(pg=3.0 from=15 to=17)): complete 2022-03-29T17:52:02.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=47, detail=PGAdvanceMap(pg=3.2 from=15 to=17)): complete 2022-03-29T17:52:02.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,322 [shard 0] osd - pg_advance_map(id=60, detail=PGAdvanceMap(pg=3.1a from=15 to=17)): complete 2022-03-29T17:52:02.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_advance_map(id=62, detail=PGAdvanceMap(pg=3.1e from=15 to=17)): complete 2022-03-29T17:52:02.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:13 2022-03-29T17:52:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active advmap 2022-03-29T17:52:02.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Started advmap 2022-03-29T17:52:02.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 16 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:01,323 [shard 0] osd - osd.2: now active 2022-03-29T17:52:02.644 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.645 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,323 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.646 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.646 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.646 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.646 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 16 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 16 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:13 2022-03-29T17:52:02.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,324 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 16 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map 2022-03-29T17:52:02.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: handle ActMap primary 2022-03-29T17:52:02.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean publish_stats_to_osd 16:16 2022-03-29T17:52:02.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,325 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 16 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 16 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,326 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 16 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,327 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.682 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 16 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean state: Active advmap 2022-03-29T17:52:02.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean state: Started advmap 2022-03-29T17:52:02.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,328 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.689 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.689 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.689 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.689 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,329 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 16 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:02.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.697 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:13 2022-03-29T17:52:02.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 16 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 16 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:02.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,331 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_advance_map(id=126, detail=PGAdvanceMap(pg=2.7 from=16 to=17)): complete 2022-03-29T17:52:02.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,332 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:13 2022-03-29T17:52:02.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean activate_map 2022-03-29T17:52:02.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean state: handle ActMap primary 2022-03-29T17:52:02.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean activate_map: Not dirtying info: last_persisted is 16 while current is 17 2022-03-29T17:52:02.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:13 2022-03-29T17:52:02.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 16:10 2022-03-29T17:52:02.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,334 [shard 0] osd - pg_advance_map(id=151, detail=PGAdvanceMap(pg=3.1e from=16 to=17)): complete 2022-03-29T17:52:02.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=149, detail=PGAdvanceMap(pg=3.1b from=16 to=17)): complete 2022-03-29T17:52:02.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=142, detail=PGAdvanceMap(pg=3.13 from=16 to=17)): complete 2022-03-29T17:52:02.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=138, detail=PGAdvanceMap(pg=3.f from=16 to=17)): complete 2022-03-29T17:52:02.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=128, detail=PGAdvanceMap(pg=3.2 from=16 to=17)): complete 2022-03-29T17:52:02.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=123, detail=PGAdvanceMap(pg=2.4 from=16 to=17)): complete 2022-03-29T17:52:02.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=145, detail=PGAdvanceMap(pg=3.16 from=16 to=17)): complete 2022-03-29T17:52:02.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=120, detail=PGAdvanceMap(pg=2.1 from=16 to=17)): complete 2022-03-29T17:52:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=121, detail=PGAdvanceMap(pg=2.2 from=16 to=17)): complete 2022-03-29T17:52:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=118, detail=PGAdvanceMap(pg=1.0 from=16 to=17)): complete 2022-03-29T17:52:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=129, detail=PGAdvanceMap(pg=3.4 from=16 to=17)): complete 2022-03-29T17:52:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=133, detail=PGAdvanceMap(pg=3.8 from=16 to=17)): complete 2022-03-29T17:52:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=144, detail=PGAdvanceMap(pg=3.15 from=16 to=17)): complete 2022-03-29T17:52:02.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=131, detail=PGAdvanceMap(pg=3.6 from=16 to=17)): complete 2022-03-29T17:52:02.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=147, detail=PGAdvanceMap(pg=3.19 from=16 to=17)): complete 2022-03-29T17:52:02.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=143, detail=PGAdvanceMap(pg=3.14 from=16 to=17)): complete 2022-03-29T17:52:02.724 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=139, detail=PGAdvanceMap(pg=3.10 from=16 to=17)): complete 2022-03-29T17:52:02.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=136, detail=PGAdvanceMap(pg=3.c from=16 to=17)): complete 2022-03-29T17:52:02.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=152, detail=PGAdvanceMap(pg=3.1f from=16 to=17)): complete 2022-03-29T17:52:02.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=132, detail=PGAdvanceMap(pg=3.7 from=16 to=17)): complete 2022-03-29T17:52:02.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=150, detail=PGAdvanceMap(pg=3.1c from=16 to=17)): complete 2022-03-29T17:52:02.725 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=146, detail=PGAdvanceMap(pg=3.17 from=16 to=17)): complete 2022-03-29T17:52:02.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=141, detail=PGAdvanceMap(pg=3.12 from=16 to=17)): complete 2022-03-29T17:52:02.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=125, detail=PGAdvanceMap(pg=2.6 from=16 to=17)): complete 2022-03-29T17:52:02.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=124, detail=PGAdvanceMap(pg=2.5 from=16 to=17)): complete 2022-03-29T17:52:02.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,335 [shard 0] osd - pg_advance_map(id=119, detail=PGAdvanceMap(pg=2.0 from=16 to=17)): complete 2022-03-29T17:52:02.726 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - pg_advance_map(id=135, detail=PGAdvanceMap(pg=3.b from=16 to=17)): complete 2022-03-29T17:52:02.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - pg_advance_map(id=134, detail=PGAdvanceMap(pg=3.a from=16 to=17)): complete 2022-03-29T17:52:02.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - pg_advance_map(id=130, detail=PGAdvanceMap(pg=3.5 from=16 to=17)): complete 2022-03-29T17:52:02.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - pg_advance_map(id=137, detail=PGAdvanceMap(pg=3.d from=16 to=17)): complete 2022-03-29T17:52:02.727 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - pg_advance_map(id=122, detail=PGAdvanceMap(pg=2.3 from=16 to=17)): complete 2022-03-29T17:52:02.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - pg_advance_map(id=148, detail=PGAdvanceMap(pg=3.1a from=16 to=17)): complete 2022-03-29T17:52:02.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - pg_advance_map(id=140, detail=PGAdvanceMap(pg=3.11 from=16 to=17)): complete 2022-03-29T17:52:02.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - pg_advance_map(id=127, detail=PGAdvanceMap(pg=3.0 from=16 to=17)): complete 2022-03-29T17:52:02.728 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,336 [shard 0] osd - osd.1: now active 2022-03-29T17:52:02.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - client_request(id=144, detail=m=[osd_op(client.4179.0:7 3.1a 3.ac0e573a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - OpSequencer::start_op: op=144, last_started=143, last_unblocked=143, last_completed=143 2022-03-29T17:52:02.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - client_request(id=144, detail=m=[osd_op(client.4179.0:7 3.1a 3:5cea7035:::rbd_info:head {write 0~19 in=19b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.729 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:02.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op write 2022-03-29T17:52:02.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - handling op write on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,336 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:02.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head submitting txn 2022-03-29T17:52:02.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean calc_trim_to_aggressive limit = 16'1 2022-03-29T17:52:02.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=16'1 mlcod 16'1 active+clean append_log log((0'0,16'1], crt=16'1) {17'2 (16'1) modify 3:5cea7035:::rbd_info:head by client.4179.0:7 2022-03-29T17:52:01.306808+0000 0 ObjectCleanRegions clean_offsets: [19~18446744073709551596], clean_omap: 1, new_object: 0} 2022-03-29T17:52:02.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - pg_epoch 17 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 luod=16'1 lua=0'0 crt=16'1 lcod 16'1 mlcod 16'1 active+clean add_log_entry 17'2 (16'1) modify 3:5cea7035:::rbd_info:head by client.4179.0:7 2022-03-29T17:52:01.306808+0000 0 ObjectCleanRegions clean_offsets: [19~18446744073709551596], clean_omap: 1, new_object: 0 2022-03-29T17:52:02.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - pg_epoch 17 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 luod=16'1 lua=0'0 crt=17'2 lcod 16'1 mlcod 16'1 active+clean append_log approx pg log length = 2 2022-03-29T17:52:02.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - pg_epoch 17 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 luod=16'1 lua=0'0 crt=17'2 lcod 16'1 mlcod 16'1 active+clean append_log transaction_applied = 1 2022-03-29T17:52:02.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,337 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:02.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,338 [shard 0] osd - replicated_request(id=2, detail=RepRequest(from=1 req=osd_repop(client.4179.0:7 3.1a e17/14 3:5cea7035:::rbd_info:head v 17'2, mlcod=17'2) v3)) start 2022-03-29T17:52:02.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,338 [shard 0] osd - pg_epoch 17 pg[3.1a( v 16'1 (0'0,16'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 mlcod 16'1 active append_log log((0'0,16'1], crt=16'1) {17'2 (16'1) modify 3:5cea7035:::rbd_info:head by client.4179.0:7 2022-03-29T17:52:01.306808+0000 0 ObjectCleanRegions clean_offsets: [19~18446744073709551596], clean_omap: 1, new_object: 0} 2022-03-29T17:52:02.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,338 [shard 0] osd - pg_epoch 17 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=16'1 lcod 16'1 mlcod 16'1 active add_log_entry 17'2 (16'1) modify 3:5cea7035:::rbd_info:head by client.4179.0:7 2022-03-29T17:52:01.306808+0000 0 ObjectCleanRegions clean_offsets: [19~18446744073709551596], clean_omap: 1, new_object: 0 2022-03-29T17:52:02.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,338 [shard 0] osd - pg_epoch 17 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 lcod 16'1 mlcod 16'1 active append_log approx pg log length = 2 2022-03-29T17:52:02.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,338 [shard 0] osd - pg_epoch 17 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 lcod 16'1 mlcod 16'1 active append_log transaction_applied = 1 2022-03-29T17:52:02.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,338 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,339 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.736 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,339 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:02.736 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,379 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:7 3.1a 3:5cea7035:::rbd_info:head {write 0~19 in=19b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:02.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,380 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,380 [shard 0] osd - OpSequencer::finish_op_in_order: op=144, last_started=144, last_unblocked=144, last_completed=143 2022-03-29T17:52:02.737 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,380 [shard 0] osd - client_request(id=144, detail=m=[osd_op(client.4179.0:7 3.1a 3:5cea7035:::rbd_info:head {write 0~19 in=19b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,381 [shard 0] osd - client_request(id=145, detail=m=[osd_op(client.4179.0:8 3.1a 3.4eeabbda (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,381 [shard 0] osd - client_request(id=145, detail=m=[osd_op(client.4179.0:8 3.1a 3.4eeabbda (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,381 [shard 0] osd - client_request(id=145, detail=m=[osd_op(client.4179.0:8 3.1a 3.4eeabbda (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,381 [shard 0] osd - OpSequencer::start_op: op=145, last_started=144, last_unblocked=144, last_completed=144 2022-03-29T17:52:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,381 [shard 0] osd - do_recover_missing check for recovery, 3:5bdd5772:::image1.rbd:head 2022-03-29T17:52:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,381 [shard 0] osd - client_request(id=145, detail=m=[osd_op(client.4179.0:8 3.1a 3:5bdd5772:::image1.rbd:head {stat} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,381 [shard 0] osd - with_head_obc 3:5bdd5772:::image1.rbd:head 2022-03-29T17:52:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,381 [shard 0] osd - with_head_obc: cache miss on 3:5bdd5772:::image1.rbd:head 2022-03-29T17:52:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - load_metadata: object 3:5bdd5772:::image1.rbd:head doesn't exist, returning empty metadata 2022-03-29T17:52:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - load_head_obc: loaded obs 3:5bdd5772:::image1.rbd:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:5bdd5772:::image1.rbd:head 2022-03-29T17:52:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - load_head_obc: returning obc 3:5bdd5772:::image1.rbd:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:5bdd5772:::image1.rbd:head 2022-03-29T17:52:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - do_osd_ops_execute: object 3:5bdd5772:::image1.rbd:head - handling op stat 2022-03-29T17:52:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - handling op stat on object 3:5bdd5772:::image1.rbd:head 2022-03-29T17:52:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - stat object does not exist 2022-03-29T17:52:02.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - rollback_obc_if_modified: object 3:5bdd5772:::image1.rbd:head got error generic:2, need_rollback=false 2022-03-29T17:52:02.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - with_head_obc: released 3:5bdd5772:::image1.rbd:head 2022-03-29T17:52:02.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - OpSequencer::finish_op_in_order: op=145, last_started=145, last_unblocked=145, last_completed=144 2022-03-29T17:52:02.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,382 [shard 0] osd - client_request(id=145, detail=m=[osd_op(client.4179.0:8 3.1a 3:5bdd5772:::image1.rbd:head {stat} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.783 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,383 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> unknown.? -@36892] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:36892/0 2022-03-29T17:52:02.783 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,383 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> client.? -@36892] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:52:02.783 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,384 [shard 0] monc - added challenge on [osd.0(client) v2:172.21.15.138:6805/35577 >> client.? -@36892] 2022-03-29T17:52:02.783 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,384 [shard 0] osd - handle_authentication client.admin has caps allow * 2022-03-29T17:52:02.784 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,385 [shard 0] ms - [osd.0(client) v2:172.21.15.138:6805/35577 >> client.4179 172.21.15.138:0/2702680834@36892] established: gs=0, pgs=4, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:52:02.784 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,385 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:9 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.784 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,385 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:9 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.784 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,385 [shard 0] osd - evt epoch is 17, i have 15, will wait 2022-03-29T17:52:02.785 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,385 [shard 0] osd - osdmap_subscribe(15) 2022-03-29T17:52:02.785 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,386 [shard 0] osd - handle_osd_map osd_map(16..17 src has 1..17) v4 2022-03-29T17:52:02.785 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,386 [shard 0] osd - handle_osd_map epochs [16..17], i have 15, src has [1..17] 2022-03-29T17:52:02.785 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,387 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:02.786 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,388 [shard 0] osd - osd.0: committed_osd_maps(16, 17) 2022-03-29T17:52:02.786 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,388 [shard 0] osd - osd.0: map e 17 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:02.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=60, detail=PGAdvanceMap(pg=1.0 from=15 to=17)): start 2022-03-29T17:52:02.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=61, detail=PGAdvanceMap(pg=2.2 from=15 to=17)): start 2022-03-29T17:52:02.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=62, detail=PGAdvanceMap(pg=2.4 from=15 to=17)): start 2022-03-29T17:52:02.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=63, detail=PGAdvanceMap(pg=2.5 from=15 to=17)): start 2022-03-29T17:52:02.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=64, detail=PGAdvanceMap(pg=2.6 from=15 to=17)): start 2022-03-29T17:52:02.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=65, detail=PGAdvanceMap(pg=2.7 from=15 to=17)): start 2022-03-29T17:52:02.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=66, detail=PGAdvanceMap(pg=3.1 from=15 to=17)): start 2022-03-29T17:52:02.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=67, detail=PGAdvanceMap(pg=3.3 from=15 to=17)): start 2022-03-29T17:52:02.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=68, detail=PGAdvanceMap(pg=3.6 from=15 to=17)): start 2022-03-29T17:52:02.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=69, detail=PGAdvanceMap(pg=3.9 from=15 to=17)): start 2022-03-29T17:52:02.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=70, detail=PGAdvanceMap(pg=3.a from=15 to=17)): start 2022-03-29T17:52:02.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=71, detail=PGAdvanceMap(pg=3.b from=15 to=17)): start 2022-03-29T17:52:02.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=72, detail=PGAdvanceMap(pg=3.c from=15 to=17)): start 2022-03-29T17:52:02.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=73, detail=PGAdvanceMap(pg=3.d from=15 to=17)): start 2022-03-29T17:52:02.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=74, detail=PGAdvanceMap(pg=3.e from=15 to=17)): start 2022-03-29T17:52:02.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=75, detail=PGAdvanceMap(pg=3.f from=15 to=17)): start 2022-03-29T17:52:02.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=76, detail=PGAdvanceMap(pg=3.10 from=15 to=17)): start 2022-03-29T17:52:02.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=77, detail=PGAdvanceMap(pg=3.12 from=15 to=17)): start 2022-03-29T17:52:02.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=78, detail=PGAdvanceMap(pg=3.13 from=15 to=17)): start 2022-03-29T17:52:02.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=79, detail=PGAdvanceMap(pg=3.15 from=15 to=17)): start 2022-03-29T17:52:02.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=80, detail=PGAdvanceMap(pg=3.17 from=15 to=17)): start 2022-03-29T17:52:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=81, detail=PGAdvanceMap(pg=3.18 from=15 to=17)): start 2022-03-29T17:52:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=82, detail=PGAdvanceMap(pg=3.1b from=15 to=17)): start 2022-03-29T17:52:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=83, detail=PGAdvanceMap(pg=3.1c from=15 to=17)): start 2022-03-29T17:52:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=84, detail=PGAdvanceMap(pg=3.1d from=15 to=17)): start 2022-03-29T17:52:02.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,389 [shard 0] osd - pg_advance_map(id=85, detail=PGAdvanceMap(pg=3.1f from=15 to=17)): start 2022-03-29T17:52:02.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 15 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 15 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 15 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 15 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 15 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,390 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.798 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 15 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.798 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.798 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.798 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 15 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.798 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active state: Started advmap 2022-03-29T17:52:02.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.799 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 15 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.800 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 15 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.801 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.802 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.803 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 15 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.803 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.803 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.803 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 15 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,391 [shard 0] osd - pg_epoch 16 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.804 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 15 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.805 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.806 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.807 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.808 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.809 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 15 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 16 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.810 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,392 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active state: Started advmap 2022-03-29T17:52:02.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.811 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.812 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:02.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.813 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.814 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 15 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 15 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.815 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.816 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.817 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,393 [shard 0] osd - pg_epoch 16 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.818 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 15 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.819 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 15 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.820 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.823 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 15 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.824 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map 2022-03-29T17:52:02.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.825 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.826 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 15 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.827 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,394 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.828 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.829 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 16 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.830 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 15 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.834 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:12 2022-03-29T17:52:02.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.835 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 15 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.836 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,395 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.837 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.838 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 15 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.839 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.840 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 16 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.841 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.842 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.843 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 16 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.844 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,396 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.845 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 15 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 16 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.848 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 15 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 16 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.849 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 16 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.850 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 16 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.850 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 16 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.851 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 15 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,397 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 16 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 16 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 16 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.859 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.860 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,398 [shard 0] osd - pg_epoch 16 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:02.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 16 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.862 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.863 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.864 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.865 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.866 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.867 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.868 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 16 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:02.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:02.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,399 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:02.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.869 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.870 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 17 2022-03-29T17:52:02.871 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:02.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.875 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 16 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:02.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:02.876 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:02.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,411 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:02.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:02.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:02.877 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:02.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:02.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 15:9 2022-03-29T17:52:02.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 17 2022-03-29T17:52:02.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - log is not dirty 2022-03-29T17:52:02.878 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:02.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=84, detail=PGAdvanceMap(pg=3.1d from=15 to=17)): complete 2022-03-29T17:52:02.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=66, detail=PGAdvanceMap(pg=3.1 from=15 to=17)): complete 2022-03-29T17:52:02.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=60, detail=PGAdvanceMap(pg=1.0 from=15 to=17)): complete 2022-03-29T17:52:02.879 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=64, detail=PGAdvanceMap(pg=2.6 from=15 to=17)): complete 2022-03-29T17:52:02.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=77, detail=PGAdvanceMap(pg=3.12 from=15 to=17)): complete 2022-03-29T17:52:02.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=79, detail=PGAdvanceMap(pg=3.15 from=15 to=17)): complete 2022-03-29T17:52:02.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=69, detail=PGAdvanceMap(pg=3.9 from=15 to=17)): complete 2022-03-29T17:52:02.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=67, detail=PGAdvanceMap(pg=3.3 from=15 to=17)): complete 2022-03-29T17:52:02.880 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=68, detail=PGAdvanceMap(pg=3.6 from=15 to=17)): complete 2022-03-29T17:52:02.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=83, detail=PGAdvanceMap(pg=3.1c from=15 to=17)): complete 2022-03-29T17:52:02.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,412 [shard 0] osd - pg_advance_map(id=70, detail=PGAdvanceMap(pg=3.a from=15 to=17)): complete 2022-03-29T17:52:02.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=75, detail=PGAdvanceMap(pg=3.f from=15 to=17)): complete 2022-03-29T17:52:02.881 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=72, detail=PGAdvanceMap(pg=3.c from=15 to=17)): complete 2022-03-29T17:52:02.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=61, detail=PGAdvanceMap(pg=2.2 from=15 to=17)): complete 2022-03-29T17:52:02.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=62, detail=PGAdvanceMap(pg=2.4 from=15 to=17)): complete 2022-03-29T17:52:02.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=73, detail=PGAdvanceMap(pg=3.d from=15 to=17)): complete 2022-03-29T17:52:02.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=71, detail=PGAdvanceMap(pg=3.b from=15 to=17)): complete 2022-03-29T17:52:02.882 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=82, detail=PGAdvanceMap(pg=3.1b from=15 to=17)): complete 2022-03-29T17:52:02.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=63, detail=PGAdvanceMap(pg=2.5 from=15 to=17)): complete 2022-03-29T17:52:02.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=76, detail=PGAdvanceMap(pg=3.10 from=15 to=17)): complete 2022-03-29T17:52:02.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=78, detail=PGAdvanceMap(pg=3.13 from=15 to=17)): complete 2022-03-29T17:52:02.883 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=74, detail=PGAdvanceMap(pg=3.e from=15 to=17)): complete 2022-03-29T17:52:02.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=65, detail=PGAdvanceMap(pg=2.7 from=15 to=17)): complete 2022-03-29T17:52:02.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=81, detail=PGAdvanceMap(pg=3.18 from=15 to=17)): complete 2022-03-29T17:52:02.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=80, detail=PGAdvanceMap(pg=3.17 from=15 to=17)): complete 2022-03-29T17:52:02.884 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - pg_advance_map(id=85, detail=PGAdvanceMap(pg=3.1f from=15 to=17)): complete 2022-03-29T17:52:02.884 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,413 [shard 0] osd - osd.0: now active 2022-03-29T17:52:02.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:9 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - OpSequencer::start_op: op=3, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:02.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - do_recover_missing check for recovery, 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.885 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:9 3.17 3:ece625d0:::rbd_id.image1:head {stat} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - with_head_obc 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,413 [shard 0] osd - with_head_obc: cache miss on 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - load_metadata: object 3:ece625d0:::rbd_id.image1:head doesn't exist, returning empty metadata 2022-03-29T17:52:02.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - load_head_obc: loaded obs 3:ece625d0:::rbd_id.image1:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.886 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - load_head_obc: returning obc 3:ece625d0:::rbd_id.image1:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head - handling op stat 2022-03-29T17:52:02.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - handling op stat on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - stat object does not exist 2022-03-29T17:52:02.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - rollback_obc_if_modified: object 3:ece625d0:::rbd_id.image1:head got error generic:2, need_rollback=false 2022-03-29T17:52:02.887 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - with_head_obc: released 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - OpSequencer::finish_op_in_order: op=3, last_started=3, last_unblocked=3, last_completed=0 2022-03-29T17:52:02.888 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,414 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:9 3.17 3:ece625d0:::rbd_id.image1:head {stat} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,417 [shard 0] osd - client_request(id=146, detail=m=[osd_op(client.4179.0:10 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,417 [shard 0] osd - client_request(id=146, detail=m=[osd_op(client.4179.0:10 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,417 [shard 0] osd - client_request(id=146, detail=m=[osd_op(client.4179.0:10 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,417 [shard 0] osd - OpSequencer::start_op: op=146, last_started=145, last_unblocked=145, last_completed=145 2022-03-29T17:52:02.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - client_request(id=146, detail=m=[osd_op(client.4179.0:10 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:02.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op call 2022-03-29T17:52:02.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - handling op call on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:02.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - handling op omap-get-vals on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:02.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,418 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:02.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:10 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:02.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - OpSequencer::finish_op_in_order: op=146, last_started=146, last_unblocked=146, last_completed=145 2022-03-29T17:52:02.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - client_request(id=146, detail=m=[osd_op(client.4179.0:10 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - client_request(id=147, detail=m=[osd_op(client.4179.0:11 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - client_request(id=147, detail=m=[osd_op(client.4179.0:11 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - client_request(id=147, detail=m=[osd_op(client.4179.0:11 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - OpSequencer::start_op: op=147, last_started=146, last_unblocked=146, last_completed=146 2022-03-29T17:52:02.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - client_request(id=147, detail=m=[osd_op(client.4179.0:11 3.1a 3:5cea7035:::rbd_info:head {read 0~0} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,419 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:02.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op read 2022-03-29T17:52:02.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - handling op read on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - read: data length: 19 2022-03-29T17:52:02.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:02.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:11 3.1a 3:5cea7035:::rbd_info:head {read 0~0 out=19b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:02.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:02.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - OpSequencer::finish_op_in_order: op=147, last_started=147, last_unblocked=147, last_completed=146 2022-03-29T17:52:02.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - client_request(id=147, detail=m=[osd_op(client.4179.0:11 3.1a 3:5cea7035:::rbd_info:head {read 0~0 out=19b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.902 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - client_request(id=148, detail=m=[osd_op(client.4179.0:12 3.1c 3.30a98c1c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,420 [shard 0] osd - client_request(id=148, detail=m=[osd_op(client.4179.0:12 3.1c 3.30a98c1c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - client_request(id=148, detail=m=[osd_op(client.4179.0:12 3.1c 3.30a98c1c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - OpSequencer::start_op: op=148, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:02.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - do_recover_missing check for recovery, 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - client_request(id=148, detail=m=[osd_op(client.4179.0:12 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_add_image in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - with_head_obc 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - with_head_obc: cache miss on 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - load_metadata: object 3:3831950c:::rbd_directory:head doesn't exist, returning empty metadata 2022-03-29T17:52:02.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - load_head_obc: loaded obs 3:3831950c:::rbd_directory:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - load_head_obc: returning obc 3:3831950c:::rbd_directory:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head - handling op call 2022-03-29T17:52:02.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - handling op call on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,421 [shard 0] osd - calling method rbd.dir_add_image, num_read=0, num_write=0 2022-03-29T17:52:02.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - handling op create on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - handling op omap-set-vals on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:52:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head all operations successful 2022-03-29T17:52:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head submitting txn 2022-03-29T17:52:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:3831950c:::rbd_directory:head by client.4179.0:12 2022-03-29T17:52:01.420609+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:02.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - pg_epoch 17 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 17'1 (0'0) modify 3:3831950c:::rbd_directory:head by client.4179.0:12 2022-03-29T17:52:01.420609+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:02.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - pg_epoch 17 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:02.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - pg_epoch 17 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:02.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,422 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:02.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,423 [shard 0] osd - replicated_request(id=98, detail=RepRequest(from=1 req=osd_repop(client.4179.0:12 3.1c e17/14 3:3831950c:::rbd_directory:head v 17'1, mlcod=17'1) v3)) start 2022-03-29T17:52:02.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,424 [shard 0] osd - pg_epoch 17 pg[3.1c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:3831950c:::rbd_directory:head by client.4179.0:12 2022-03-29T17:52:01.420609+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:02.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,424 [shard 0] osd - pg_epoch 17 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 17'1 (0'0) modify 3:3831950c:::rbd_directory:head by client.4179.0:12 2022-03-29T17:52:01.420609+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:02.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,425 [shard 0] osd - pg_epoch 17 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:02.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,425 [shard 0] osd - pg_epoch 17 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:02.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,425 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,425 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,425 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:02.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,465 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:12 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_add_image in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:3831950c:::rbd_directory:head sending reply 2022-03-29T17:52:02.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,466 [shard 0] osd - with_head_obc: released 3:3831950c:::rbd_directory:head 2022-03-29T17:52:02.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,466 [shard 0] osd - OpSequencer::finish_op_in_order: op=148, last_started=148, last_unblocked=148, last_completed=0 2022-03-29T17:52:02.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,466 [shard 0] osd - client_request(id=148, detail=m=[osd_op(client.4179.0:12 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_add_image in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,466 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:13 3.17 3.ba46737 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,466 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:13 3.17 3.ba46737 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,466 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:13 3.17 3.ba46737 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,466 [shard 0] osd - OpSequencer::start_op: op=4, last_started=3, last_unblocked=3, last_completed=3 2022-03-29T17:52:02.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - do_recover_missing check for recovery, 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:13 3.17 3:ece625d0:::rbd_id.image1:head {create, call rbd.set_id in=25b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - with_head_obc 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - with_head_obc: found 3:ece625d0:::rbd_id.image1:head in cache 2022-03-29T17:52:02.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head - handling op create 2022-03-29T17:52:02.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - handling op create on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head - handling op call 2022-03-29T17:52:02.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - handling op call on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - calling method rbd.set_id, num_read=0, num_write=1 2022-03-29T17:52:02.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - handling op stat on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,467 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=0.0 2022-03-29T17:52:02.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - handling op stat on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=0.0 2022-03-29T17:52:02.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - handling op write on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=0, num_write=2 2022-03-29T17:52:02.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head all operations successful 2022-03-29T17:52:02.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head submitting txn 2022-03-29T17:52:02.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:02.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:ece625d0:::rbd_id.image1:head by client.4179.0:13 2022-03-29T17:52:01.466157+0000 0 ObjectCleanRegions clean_offsets: [16~18446744073709551599], clean_omap: 1, new_object: 0} 2022-03-29T17:52:02.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - pg_epoch 17 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 17'1 (0'0) modify 3:ece625d0:::rbd_id.image1:head by client.4179.0:13 2022-03-29T17:52:01.466157+0000 0 ObjectCleanRegions clean_offsets: [16~18446744073709551599], clean_omap: 1, new_object: 0 2022-03-29T17:52:02.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - pg_epoch 17 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:02.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - pg_epoch 17 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:02.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,468 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:02.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,469 [shard 0] osd - replicated_request(id=1, detail=RepRequest(from=0 req=osd_repop(client.4179.0:13 3.17 e17/14 3:ece625d0:::rbd_id.image1:head v 17'1, mlcod=17'1) v3)) start 2022-03-29T17:52:02.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,470 [shard 0] osd - pg_epoch 17 pg[3.17( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:ece625d0:::rbd_id.image1:head by client.4179.0:13 2022-03-29T17:52:01.466157+0000 0 ObjectCleanRegions clean_offsets: [16~18446744073709551599], clean_omap: 1, new_object: 0} 2022-03-29T17:52:02.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,470 [shard 0] osd - pg_epoch 17 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 17'1 (0'0) modify 3:ece625d0:::rbd_id.image1:head by client.4179.0:13 2022-03-29T17:52:01.466157+0000 0 ObjectCleanRegions clean_offsets: [16~18446744073709551599], clean_omap: 1, new_object: 0 2022-03-29T17:52:02.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,470 [shard 0] osd - pg_epoch 17 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:02.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,470 [shard 0] osd - pg_epoch 17 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:02.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,470 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,470 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,470 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:02.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,510 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:13 3.17 3:ece625d0:::rbd_id.image1:head {create, call rbd.set_id in=25b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:ece625d0:::rbd_id.image1:head sending reply 2022-03-29T17:52:02.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,511 [shard 0] osd - with_head_obc: released 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,511 [shard 0] osd - OpSequencer::finish_op_in_order: op=4, last_started=4, last_unblocked=4, last_completed=3 2022-03-29T17:52:02.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,511 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:13 3.17 3:ece625d0:::rbd_id.image1:head {create, call rbd.set_id in=25b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,511 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:14 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,511 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:14 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,511 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:14 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,511 [shard 0] osd - OpSequencer::start_op: op=5, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:02.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:14 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {create, call rbd.create in=59b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - with_head_obc: cache miss on 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - load_metadata: object 3:a9e8038f:::rbd_header.1053953fed53:head doesn't exist, returning empty metadata 2022-03-29T17:52:02.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - load_head_obc: loaded obs 3:a9e8038f:::rbd_header.1053953fed53:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - load_head_obc: returning obc 3:a9e8038f:::rbd_header.1053953fed53:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op create 2022-03-29T17:52:02.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - handling op create on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:02.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.941 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,512 [shard 0] osd - calling method rbd.create, num_read=0, num_write=1 2022-03-29T17:52:02.941 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.941 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - handling op omap-set-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.941 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=2 2022-03-29T17:52:02.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:02.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head submitting txn 2022-03-29T17:52:02.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:02.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:14 2022-03-29T17:52:01.511520+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:02.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 17'1 (0'0) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:14 2022-03-29T17:52:01.511520+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:02.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:02.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:02.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,513 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.944 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,514 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:02.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,551 [shard 0] osd - replicated_request(id=2, detail=RepRequest(from=0 req=osd_repop(client.4179.0:14 3.15 e17/14 3:a9e8038f:::rbd_header.1053953fed53:head v 17'1, mlcod=17'1) v3)) start 2022-03-29T17:52:02.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,552 [shard 0] osd - pg_epoch 17 pg[3.15( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:14 2022-03-29T17:52:01.511520+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:02.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,552 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 17'1 (0'0) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:14 2022-03-29T17:52:01.511520+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:02.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,552 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:02.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,552 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:02.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,552 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,552 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,552 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:02.947 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,592 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:14 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {create, call rbd.create in=59b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:02.947 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,593 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.948 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,593 [shard 0] osd - OpSequencer::finish_op_in_order: op=5, last_started=5, last_unblocked=5, last_completed=0 2022-03-29T17:52:02.948 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,593 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:14 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {create, call rbd.create in=59b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,593 [shard 0] osd - client_request(id=149, detail=m=[osd_op(client.4179.0:15 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,593 [shard 0] osd - client_request(id=149, detail=m=[osd_op(client.4179.0:15 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - client_request(id=149, detail=m=[osd_op(client.4179.0:15 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - OpSequencer::start_op: op=149, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:02.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - do_recover_missing check for recovery, 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - client_request(id=149, detail=m=[osd_op(client.4179.0:15 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call rbd.object_map_resize in=29b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - with_head_obc 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - with_head_obc: cache miss on 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - load_metadata: object 3:b2b71b89:::rbd_object_map.1053953fed53:head doesn't exist, returning empty metadata 2022-03-29T17:52:02.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - load_head_obc: loaded obs 3:b2b71b89:::rbd_object_map.1053953fed53:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - load_head_obc: returning obc 3:b2b71b89:::rbd_object_map.1053953fed53:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head - handling op call 2022-03-29T17:52:02.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - handling op call on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,594 [shard 0] osd - calling method rbd.object_map_resize, num_read=0, num_write=0 2022-03-29T17:52:02.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - handling op stat on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - stat object does not exist 2022-03-29T17:52:02.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - handling op writefull on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=0, num_write=1 2022-03-29T17:52:02.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head all operations successful 2022-03-29T17:52:02.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head submitting txn 2022-03-29T17:52:02.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:02.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:15 2022-03-29T17:52:01.593388+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:02.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 17'1 (0'0) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:15 2022-03-29T17:52:01.593388+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:02.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:02.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:02.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,595 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,596 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:02.958 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,596 [shard 0] osd - replicated_request(id=99, detail=RepRequest(from=1 req=osd_repop(client.4179.0:15 3.d e17/14 3:b2b71b89:::rbd_object_map.1053953fed53:head v 17'1, mlcod=17'1) v3)) start 2022-03-29T17:52:02.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,597 [shard 0] osd - pg_epoch 17 pg[3.d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:15 2022-03-29T17:52:01.593388+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:02.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,597 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 17'1 (0'0) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:15 2022-03-29T17:52:01.593388+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:02.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,597 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:02.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,597 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:02.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,597 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:02.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,597 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:02.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,597 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:02.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,637 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:15 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call rbd.object_map_resize in=29b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:b2b71b89:::rbd_object_map.1053953fed53:head sending reply 2022-03-29T17:52:02.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,637 [shard 0] osd - with_head_obc: released 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:02.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,638 [shard 0] osd - OpSequencer::finish_op_in_order: op=149, last_started=149, last_unblocked=149, last_completed=0 2022-03-29T17:52:02.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,638 [shard 0] osd - client_request(id=149, detail=m=[osd_op(client.4179.0:15 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call rbd.object_map_resize in=29b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.982 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:16 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.982 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:16 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.982 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:16 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.983 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - OpSequencer::start_op: op=6, last_started=4, last_unblocked=4, last_completed=4 2022-03-29T17:52:02.983 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - do_recover_missing check for recovery, 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.983 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:16 3.17 3:ece625d0:::rbd_id.image1:head {stat} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.983 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - with_head_obc 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.984 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - with_head_obc: found 3:ece625d0:::rbd_id.image1:head in cache 2022-03-29T17:52:02.984 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head - handling op stat 2022-03-29T17:52:02.984 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - handling op stat on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.984 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - stat os.oi.size=16, os.oi.mtime=1648576321.4661574 2022-03-29T17:52:02.984 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head all operations successful 2022-03-29T17:52:02.985 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,641 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:16 3.17 3:ece625d0:::rbd_id.image1:head {stat out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:ece625d0:::rbd_id.image1:head sending reply 2022-03-29T17:52:02.985 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - with_head_obc: released 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.985 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - OpSequencer::finish_op_in_order: op=6, last_started=6, last_unblocked=6, last_completed=4 2022-03-29T17:52:02.985 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:16 3.17 3:ece625d0:::rbd_id.image1:head {stat out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.986 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4179.0:17 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.986 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4179.0:17 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.986 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4179.0:17 3.17 3.ba46737 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.986 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - OpSequencer::start_op: op=7, last_started=6, last_unblocked=6, last_completed=6 2022-03-29T17:52:02.986 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - do_recover_missing check for recovery, 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4179.0:17 3.17 3:ece625d0:::rbd_id.image1:head {call rbd.get_id in=9b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - with_head_obc 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,642 [shard 0] osd - with_head_obc: found 3:ece625d0:::rbd_id.image1:head in cache 2022-03-29T17:52:02.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,643 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head - handling op call 2022-03-29T17:52:02.987 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,643 [shard 0] osd - handling op call on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,643 [shard 0] osd - calling method rbd.get_id, num_read=0, num_write=0 2022-03-29T17:52:02.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,643 [shard 0] osd - handling op stat on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,643 [shard 0] osd - stat os.oi.size=16, os.oi.mtime=1648576321.4661574 2022-03-29T17:52:02.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,643 [shard 0] osd - handling op sync_read on object 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.988 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - read: data length: 16 2022-03-29T17:52:02.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=1, num_write=0 2022-03-29T17:52:02.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - do_osd_ops_execute: object 3:ece625d0:::rbd_id.image1:head all operations successful 2022-03-29T17:52:02.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:17 3.17 3:ece625d0:::rbd_id.image1:head {call rbd.get_id in=9b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:ece625d0:::rbd_id.image1:head sending reply 2022-03-29T17:52:02.989 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - with_head_obc: released 3:ece625d0:::rbd_id.image1:head 2022-03-29T17:52:02.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - OpSequencer::finish_op_in_order: op=7, last_started=7, last_unblocked=7, last_completed=6 2022-03-29T17:52:02.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4179.0:17 3.17 3:ece625d0:::rbd_id.image1:head {call rbd.get_id in=9b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4179.0:18 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,644 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4179.0:18 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.990 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4179.0:18 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - OpSequencer::start_op: op=8, last_started=5, last_unblocked=5, last_completed=5 2022-03-29T17:52:02.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4179.0:18 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b, call rbd.get_object_prefix in=20b, call rbd.get_features in=24b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.991 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:02.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:02.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - calling method rbd.get_size, num_read=0, num_write=0 2022-03-29T17:52:02.992 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,645 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,646 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,646 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=9 while num_read=2, num_write=0 2022-03-29T17:52:02.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,646 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:02.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,646 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,646 [shard 0] osd - calling method rbd.get_object_prefix, num_read=2, num_write=0 2022-03-29T17:52:02.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,646 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,647 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=25 while num_read=3, num_write=0 2022-03-29T17:52:02.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,647 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:02.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,647 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,647 [shard 0] osd - calling method rbd.get_features, num_read=3, num_write=0 2022-03-29T17:52:02.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,647 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,647 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=4, num_write=0 2022-03-29T17:52:02.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:02.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:18 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_object_prefix in=20b out=25b, call rbd.get_features in=24b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:02.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - OpSequencer::finish_op_in_order: op=8, last_started=8, last_unblocked=8, last_completed=5 2022-03-29T17:52:02.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4179.0:18 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_object_prefix in=20b out=25b, call rbd.get_features in=24b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:02.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4179.0:19 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:02.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4179.0:19 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:02.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4179.0:19 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:02.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - OpSequencer::start_op: op=9, last_started=8, last_unblocked=8, last_completed=8 2022-03-29T17:52:02.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,648 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,649 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4179.0:19 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_create_timestamp in=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:02.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,649 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,649 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:02.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,649 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:02.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,649 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.998 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,649 [shard 0] osd - calling method rbd.get_create_timestamp, num_read=0, num_write=0 2022-03-29T17:52:02.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,649 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:02.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,649 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:02.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:02.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:19 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_create_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:02.999 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.000 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - OpSequencer::finish_op_in_order: op=9, last_started=9, last_unblocked=9, last_completed=8 2022-03-29T17:52:03.000 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4179.0:19 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_create_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.000 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4179.0:20 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.000 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4179.0:20 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.000 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4179.0:20 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.001 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - OpSequencer::start_op: op=10, last_started=9, last_unblocked=9, last_completed=9 2022-03-29T17:52:03.001 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,650 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.001 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4179.0:20 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_access_timestamp in=23b, call rbd.get_modify_timestamp in=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.001 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - calling method rbd.get_access_timestamp, num_read=0, num_write=0 2022-03-29T17:52:03.002 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:03.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,651 [shard 0] osd - calling method rbd.get_modify_timestamp, num_read=1, num_write=0 2022-03-29T17:52:03.003 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,652 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,652 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=2, num_write=0 2022-03-29T17:52:03.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,652 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,652 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:20 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_access_timestamp in=23b out=8b, call rbd.get_modify_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.004 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - OpSequencer::finish_op_in_order: op=10, last_started=10, last_unblocked=10, last_completed=9 2022-03-29T17:52:03.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4179.0:20 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_access_timestamp in=23b out=8b, call rbd.get_modify_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4179.0:21 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4179.0:21 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.005 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4179.0:21 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - OpSequencer::start_op: op=11, last_started=10, last_unblocked=10, last_completed=10 2022-03-29T17:52:03.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4179.0:21 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_data_pool in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.006 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,653 [shard 0] osd - calling method rbd.get_data_pool, num_read=0, num_write=0 2022-03-29T17:52:03.007 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,654 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,654 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:03.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,654 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,654 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:21 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_data_pool in=16b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,654 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.008 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,655 [shard 0] osd - OpSequencer::finish_op_in_order: op=11, last_started=11, last_unblocked=11, last_completed=10 2022-03-29T17:52:03.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,655 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4179.0:21 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_data_pool in=16b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4179.0:22 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4179.0:22 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4179.0:22 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.009 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - OpSequencer::start_op: op=12, last_started=11, last_unblocked=11, last_completed=11 2022-03-29T17:52:03.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4179.0:22 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b, call rbd.get_features in=24b, call rbd.get_flags in=20b, call rbd.get_snapcontext in=18b, call lock.get_info in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.010 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,656 [shard 0] osd - calling method rbd.get_size, num_read=0, num_write=0 2022-03-29T17:52:03.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,657 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.011 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,657 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,657 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=9 while num_read=2, num_write=0 2022-03-29T17:52:03.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,657 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,657 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,657 [shard 0] osd - calling method rbd.get_features, num_read=2, num_write=0 2022-03-29T17:52:03.012 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,658 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,658 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=3, num_write=0 2022-03-29T17:52:03.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,658 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,658 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,658 [shard 0] osd - calling method rbd.get_flags, num_read=3, num_write=0 2022-03-29T17:52:03.013 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,658 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,659 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=4, num_write=0 2022-03-29T17:52:03.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,659 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,659 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.014 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,659 [shard 0] osd - calling method rbd.get_snapcontext, num_read=4, num_write=0 2022-03-29T17:52:03.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,659 [shard 0] osd - handling op omap-get-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,660 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,660 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=12 while num_read=6, num_write=0 2022-03-29T17:52:03.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,660 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.015 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,660 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,660 [shard 0] osd - calling method lock.get_info, num_read=6, num_write=0 2022-03-29T17:52:03.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,660 [shard 0] osd - handling op getxattr on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,660 [shard 0] osd - getxattr on obj=3:a9e8038f:::rbd_header.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=15 while num_read=7, num_write=0 2022-03-29T17:52:03.016 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:22 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_features in=24b out=16b, call rbd.get_flags in=20b out=8b, call rbd.get_snapcontext in=18b out=12b, call lock.get_info in=30b out=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - OpSequencer::finish_op_in_order: op=12, last_started=12, last_unblocked=12, last_completed=11 2022-03-29T17:52:03.017 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4179.0:22 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_features in=24b out=16b, call rbd.get_flags in=20b out=8b, call rbd.get_snapcontext in=18b out=12b, call lock.get_info in=30b out=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4179.0:23 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4179.0:23 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4179.0:23 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,661 [shard 0] osd - OpSequencer::start_op: op=13, last_started=12, last_unblocked=12, last_completed=12 2022-03-29T17:52:03.018 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4179.0:23 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b, call rbd.parent_overlap_get in=29b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.019 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - calling method rbd.parent_get, num_read=0, num_write=0 2022-03-29T17:52:03.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576321.5115206 2022-03-29T17:52:03.020 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,662 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,663 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,663 [shard 0] osd - handling op omap-get-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,663 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=30 while num_read=3, num_write=0 2022-03-29T17:52:03.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,663 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.021 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,663 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,663 [shard 0] osd - calling method rbd.parent_overlap_get, num_read=3, num_write=0 2022-03-29T17:52:03.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,663 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,663 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576321.5115206 2022-03-29T17:52:03.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,664 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.022 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,664 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.023 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,664 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=1 while num_read=5, num_write=0 2022-03-29T17:52:03.023 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,664 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.023 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,664 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:23 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b out=30b, call rbd.parent_overlap_get in=29b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.023 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,664 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,664 [shard 0] osd - OpSequencer::finish_op_in_order: op=13, last_started=13, last_unblocked=13, last_completed=12 2022-03-29T17:52:03.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,664 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4179.0:23 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b out=30b, call rbd.parent_overlap_get in=29b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4179.0:24 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4179.0:24 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.024 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4179.0:24 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - OpSequencer::start_op: op=14, last_started=13, last_unblocked=13, last_completed=13 2022-03-29T17:52:03.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4179.0:24 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.025 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,665 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,666 [shard 0] osd - handling op omap-get-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.026 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,666 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,666 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,666 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:24 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,666 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.027 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,666 [shard 0] osd - OpSequencer::finish_op_in_order: op=14, last_started=14, last_unblocked=14, last_completed=13 2022-03-29T17:52:03.028 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,666 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4179.0:24 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - client_request(id=150, detail=m=[osd_op(client.4179.0:25 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - client_request(id=150, detail=m=[osd_op(client.4179.0:25 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - client_request(id=150, detail=m=[osd_op(client.4179.0:25 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - OpSequencer::start_op: op=150, last_started=147, last_unblocked=147, last_completed=147 2022-03-29T17:52:03.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - client_request(id=150, detail=m=[osd_op(client.4179.0:25 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:03.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op call 2022-03-29T17:52:03.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - handling op call on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,667 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,668 [shard 0] osd - handling op omap-get-vals on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,668 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,668 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:03.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,668 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:25 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:03.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,668 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - OpSequencer::finish_op_in_order: op=150, last_started=150, last_unblocked=150, last_completed=147 2022-03-29T17:52:03.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - client_request(id=150, detail=m=[osd_op(client.4179.0:25 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4179.0:26 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4179.0:26 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4179.0:26 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.038 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - OpSequencer::start_op: op=15, last_started=14, last_unblocked=14, last_completed=14 2022-03-29T17:52:03.039 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.039 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4179.0:26 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.039 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.039 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,669 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,670 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,670 [shard 0] osd - calling method rbd.image_group_get, num_read=0, num_write=0 2022-03-29T17:52:03.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,670 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.040 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,670 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=18 while num_read=1, num_write=0 2022-03-29T17:52:03.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,670 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,670 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:26 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b out=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - OpSequencer::finish_op_in_order: op=15, last_started=15, last_unblocked=15, last_completed=14 2022-03-29T17:52:03.041 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4179.0:26 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b out=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4179.0:27 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4179.0:27 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4179.0:27 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.042 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - OpSequencer::start_op: op=16, last_started=15, last_unblocked=15, last_completed=15 2022-03-29T17:52:03.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4179.0:27 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {watch watch cookie 140032509772432} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.043 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op watch 2022-03-29T17:52:03.044 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - handling op watch on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.044 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - do_op_watch 2022-03-29T17:52:03.044 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - do_op_watch_subop_watch 2022-03-29T17:52:03.044 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,671 [shard 0] osd - registered new watch watch(cookie 140032509772432 30s 172.21.15.138:0/2702680834) by client.4179 2022-03-29T17:52:03.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,671 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head submitting txn 2022-03-29T17:52:03.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean calc_trim_to_aggressive limit = 17'1 2022-03-29T17:52:03.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean append_log log((0'0,17'1], crt=17'1) {17'2 (17'1) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:27 2022-03-29T17:52:01.671191+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.045 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'1 lua=0'0 crt=17'1 lcod 17'1 mlcod 17'1 active+clean add_log_entry 17'2 (17'1) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:27 2022-03-29T17:52:01.671191+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'1 lua=0'0 crt=17'2 lcod 17'1 mlcod 17'1 active+clean append_log approx pg log length = 2 2022-03-29T17:52:03.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'1 lua=0'0 crt=17'2 lcod 17'1 mlcod 17'1 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.046 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.047 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,672 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,673 [shard 0] osd - replicated_request(id=3, detail=RepRequest(from=0 req=osd_repop(client.4179.0:27 3.15 e17/14 3:a9e8038f:::rbd_header.1053953fed53:head v 17'2, mlcod=17'2) v3)) start 2022-03-29T17:52:03.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,673 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active append_log log((0'0,17'1], crt=17'1) {17'2 (17'1) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:27 2022-03-29T17:52:01.671191+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,673 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 17'1 mlcod 17'1 active add_log_entry 17'2 (17'1) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:27 2022-03-29T17:52:01.671191+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,673 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 lcod 17'1 mlcod 17'1 active append_log approx pg log length = 2 2022-03-29T17:52:03.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,674 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 lcod 17'1 mlcod 17'1 active append_log transaction_applied = 1 2022-03-29T17:52:03.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,674 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,674 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,674 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.055 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,715 [shard 0] osd - op_effect: added new watcher: 140032509772432,client.4179 2022-03-29T17:52:03.055 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,715 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:27 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {watch watch cookie 140032509772432} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.055 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,715 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.055 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,715 [shard 0] osd - OpSequencer::finish_op_in_order: op=16, last_started=16, last_unblocked=16, last_completed=15 2022-03-29T17:52:03.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,715 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4179.0:27 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {watch watch cookie 140032509772432} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4179.0:28 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4179.0:28 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4179.0:28 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.056 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - OpSequencer::start_op: op=17, last_started=16, last_unblocked=16, last_completed=16 2022-03-29T17:52:03.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4179.0:28 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call lock.get_info in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.057 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,716 [shard 0] osd - calling method lock.get_info, num_read=0, num_write=0 2022-03-29T17:52:03.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - handling op getxattr on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.058 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - getxattr on obj=3:a9e8038f:::rbd_header.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.059 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=15 while num_read=1, num_write=0 2022-03-29T17:52:03.059 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.059 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:28 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call lock.get_info in=30b out=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.059 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - OpSequencer::finish_op_in_order: op=17, last_started=17, last_unblocked=17, last_completed=16 2022-03-29T17:52:03.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - client_request(id=17, detail=m=[osd_op(client.4179.0:28 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call lock.get_info in=30b out=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4179.0:29 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,717 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4179.0:29 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.060 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4179.0:29 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.061 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - OpSequencer::start_op: op=18, last_started=17, last_unblocked=17, last_completed=17 2022-03-29T17:52:03.061 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.061 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4179.0:29 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call lock.lock in=76b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.061 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.061 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.062 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.062 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.062 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:52:03.062 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - handling op getxattr on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.063 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,718 [shard 0] osd - getxattr on obj=3:a9e8038f:::rbd_header.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.063 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - handling op setxattr on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.063 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,719 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:52:03.063 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - setxattr on obj=3:a9e8038f:::rbd_header.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.063 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:52:03.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head submitting txn 2022-03-29T17:52:03.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean calc_trim_to_aggressive limit = 17'2 2022-03-29T17:52:03.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean append_log log((0'0,17'2], crt=17'2) {17'3 (17'2) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:29 2022-03-29T17:52:01.717655+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.064 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'2 lua=0'0 crt=17'2 lcod 17'2 mlcod 17'2 active+clean add_log_entry 17'3 (17'2) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:29 2022-03-29T17:52:01.717655+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.065 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'2 lua=0'0 crt=17'3 lcod 17'2 mlcod 17'2 active+clean append_log approx pg log length = 3 2022-03-29T17:52:03.065 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'2 lua=0'0 crt=17'3 lcod 17'2 mlcod 17'2 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.065 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.065 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.066 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,719 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,755 [shard 0] osd - replicated_request(id=4, detail=RepRequest(from=0 req=osd_repop(client.4179.0:29 3.15 e17/14 3:a9e8038f:::rbd_header.1053953fed53:head v 17'3, mlcod=17'3) v3)) start 2022-03-29T17:52:03.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,756 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active append_log log((0'0,17'2], crt=17'2) {17'3 (17'2) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:29 2022-03-29T17:52:01.717655+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,756 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 lcod 17'2 mlcod 17'2 active add_log_entry 17'3 (17'2) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:29 2022-03-29T17:52:01.717655+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,756 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 lcod 17'2 mlcod 17'2 active append_log approx pg log length = 3 2022-03-29T17:52:03.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,756 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 lcod 17'2 mlcod 17'2 active append_log transaction_applied = 1 2022-03-29T17:52:03.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,756 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,756 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,756 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.069 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,796 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:29 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call lock.lock in=76b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.070 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,797 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.070 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,797 [shard 0] osd - OpSequencer::finish_op_in_order: op=18, last_started=18, last_unblocked=18, last_completed=17 2022-03-29T17:52:03.070 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,797 [shard 0] osd - client_request(id=18, detail=m=[osd_op(client.4179.0:29 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call lock.lock in=76b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,797 [shard 0] osd - client_request(id=151, detail=m=[osd_op(client.4179.0:30 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,797 [shard 0] osd - client_request(id=151, detail=m=[osd_op(client.4179.0:30 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,797 [shard 0] osd - client_request(id=151, detail=m=[osd_op(client.4179.0:30 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,797 [shard 0] osd - OpSequencer::start_op: op=151, last_started=149, last_unblocked=149, last_completed=149 2022-03-29T17:52:03.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - do_recover_missing check for recovery, 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - client_request(id=151, detail=m=[osd_op(client.4179.0:30 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.lock in=48b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - with_head_obc 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - with_head_obc: found 3:b2b71b89:::rbd_object_map.1053953fed53:head in cache 2022-03-29T17:52:03.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head - handling op call 2022-03-29T17:52:03.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - handling op call on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 2022-03-29T17:52:03.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - handling op getxattr on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,798 [shard 0] osd - getxattr on obj=3:b2b71b89:::rbd_object_map.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - handling op setxattr on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,799 [shard 0] alienstore - get_max_attr_name_length 2022-03-29T17:52:03.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - setxattr on obj=3:b2b71b89:::rbd_object_map.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=1 2022-03-29T17:52:03.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head all operations successful 2022-03-29T17:52:03.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head submitting txn 2022-03-29T17:52:03.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean calc_trim_to_aggressive limit = 17'1 2022-03-29T17:52:03.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean append_log log((0'0,17'1], crt=17'1) {17'2 (17'1) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:30 2022-03-29T17:52:01.797282+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'1 lua=0'0 crt=17'1 lcod 17'1 mlcod 17'1 active+clean add_log_entry 17'2 (17'1) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:30 2022-03-29T17:52:01.797282+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'1 lua=0'0 crt=17'2 lcod 17'1 mlcod 17'1 active+clean append_log approx pg log length = 2 2022-03-29T17:52:03.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'1 lua=0'0 crt=17'2 lcod 17'1 mlcod 17'1 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,799 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,800 [shard 0] osd - replicated_request(id=100, detail=RepRequest(from=1 req=osd_repop(client.4179.0:30 3.d e17/14 3:b2b71b89:::rbd_object_map.1053953fed53:head v 17'2, mlcod=17'2) v3)) start 2022-03-29T17:52:03.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,801 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active append_log log((0'0,17'1], crt=17'1) {17'2 (17'1) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:30 2022-03-29T17:52:01.797282+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.080 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,801 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 17'1 mlcod 17'1 active add_log_entry 17'2 (17'1) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:30 2022-03-29T17:52:01.797282+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,801 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 lcod 17'1 mlcod 17'1 active append_log approx pg log length = 2 2022-03-29T17:52:03.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,801 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 lcod 17'1 mlcod 17'1 active append_log transaction_applied = 1 2022-03-29T17:52:03.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,801 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.081 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,801 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.082 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,801 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.082 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,810 [shard 0] alienstore - stat 2022-03-29T17:52:03.082 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:01,822 [shard 0] alienstore - stat 2022-03-29T17:52:03.083 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:01,826 [shard 0] alienstore - stat 2022-03-29T17:52:03.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,841 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:30 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.lock in=48b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:b2b71b89:::rbd_object_map.1053953fed53:head sending reply 2022-03-29T17:52:03.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - with_head_obc: released 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - OpSequencer::finish_op_in_order: op=151, last_started=151, last_unblocked=151, last_completed=149 2022-03-29T17:52:03.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - client_request(id=151, detail=m=[osd_op(client.4179.0:30 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.lock in=48b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - client_request(id=152, detail=m=[osd_op(client.4179.0:31 3.d 3.91d8ed4d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - client_request(id=152, detail=m=[osd_op(client.4179.0:31 3.d 3.91d8ed4d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - client_request(id=152, detail=m=[osd_op(client.4179.0:31 3.d 3.91d8ed4d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - OpSequencer::start_op: op=152, last_started=151, last_unblocked=151, last_completed=151 2022-03-29T17:52:03.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - do_recover_missing check for recovery, 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - client_request(id=152, detail=m=[osd_op(client.4179.0:31 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call rbd.object_map_load in=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - with_head_obc 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - with_head_obc: found 3:b2b71b89:::rbd_object_map.1053953fed53:head in cache 2022-03-29T17:52:03.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,842 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head - handling op call 2022-03-29T17:52:03.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,843 [shard 0] osd - handling op call on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,843 [shard 0] osd - calling method rbd.object_map_load, num_read=0, num_write=0 2022-03-29T17:52:03.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,843 [shard 0] osd - handling op stat on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,843 [shard 0] osd - stat os.oi.size=35, os.oi.mtime=1648576321.797283 2022-03-29T17:52:03.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,843 [shard 0] osd - handling op sync_read on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,843 [shard 0] osd - read: data length: 35 2022-03-29T17:52:03.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,843 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=23 while num_read=1, num_write=0 2022-03-29T17:52:03.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,843 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head all operations successful 2022-03-29T17:52:03.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,844 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:31 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call rbd.object_map_load in=18b out=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:b2b71b89:::rbd_object_map.1053953fed53:head sending reply 2022-03-29T17:52:03.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,844 [shard 0] osd - with_head_obc: released 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,844 [shard 0] osd - OpSequencer::finish_op_in_order: op=152, last_started=152, last_unblocked=152, last_completed=151 2022-03-29T17:52:03.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,844 [shard 0] osd - client_request(id=152, detail=m=[osd_op(client.4179.0:31 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call rbd.object_map_load in=18b out=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,844 [shard 0] osd - client_request(id=153, detail=m=[osd_op(client.4179.0:33 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,844 [shard 0] osd - client_request(id=153, detail=m=[osd_op(client.4179.0:33 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,844 [shard 0] osd - client_request(id=153, detail=m=[osd_op(client.4179.0:33 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,844 [shard 0] osd - OpSequencer::start_op: op=153, last_started=152, last_unblocked=152, last_completed=152 2022-03-29T17:52:03.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - do_recover_missing check for recovery, 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - client_request(id=153, detail=m=[osd_op(client.4179.0:33 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.assert_locked in=44b, call rbd.object_map_update in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - with_head_obc 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - with_head_obc: found 3:b2b71b89:::rbd_object_map.1053953fed53:head in cache 2022-03-29T17:52:03.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head - handling op call 2022-03-29T17:52:03.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - handling op call on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - calling method lock.assert_locked, num_read=0, num_write=0 2022-03-29T17:52:03.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - handling op getxattr on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,845 [shard 0] osd - getxattr on obj=3:b2b71b89:::rbd_object_map.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=0 2022-03-29T17:52:03.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head - handling op call 2022-03-29T17:52:03.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - handling op call on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - calling method rbd.object_map_update, num_read=1, num_write=0 2022-03-29T17:52:03.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - handling op stat on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - stat os.oi.size=35, os.oi.mtime=1648576321.797283 2022-03-29T17:52:03.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - handling op sync_read on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - read: data length: 18 2022-03-29T17:52:03.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,846 [shard 0] osd - handling op sync_read on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - read: data length: 4 2022-03-29T17:52:03.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - handling op sync_read on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - read: data length: 4 2022-03-29T17:52:03.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - handling op sync_read on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - read: data length: 1 2022-03-29T17:52:03.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - handling op write on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - handling op write on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=5, num_write=2 2022-03-29T17:52:03.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head all operations successful 2022-03-29T17:52:03.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head submitting txn 2022-03-29T17:52:03.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean calc_trim_to_aggressive limit = 17'2 2022-03-29T17:52:03.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean append_log log((0'0,17'2], crt=17'2) {17'3 (17'2) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:33 2022-03-29T17:52:01.844531+0000 0 ObjectCleanRegions clean_offsets: [0~18,19~12,35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'2 lua=0'0 crt=17'2 lcod 17'2 mlcod 17'2 active+clean add_log_entry 17'3 (17'2) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:33 2022-03-29T17:52:01.844531+0000 0 ObjectCleanRegions clean_offsets: [0~18,19~12,35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'2 lua=0'0 crt=17'3 lcod 17'2 mlcod 17'2 active+clean append_log approx pg log length = 3 2022-03-29T17:52:03.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'2 lua=0'0 crt=17'3 lcod 17'2 mlcod 17'2 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,847 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.107 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,848 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4179.0:32 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.107 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,848 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4179.0:32 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.108 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,849 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4179.0:32 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.108 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,849 [shard 0] osd - OpSequencer::start_op: op=19, last_started=18, last_unblocked=18, last_completed=18 2022-03-29T17:52:03.108 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,849 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.108 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,849 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4179.0:32 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644064496 in=38b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,849 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify 2022-03-29T17:52:03.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - handling op notify on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.109 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - do_op_notify, msg epoch: 17 2022-03-29T17:52:03.110 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.110 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,850 [shard 0] osd - start_notify adding notify(id=73014444032) 2022-03-29T17:52:03.110 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,850 [shard 0] osd - send_notify_msg for notify(id=73014444032) 2022-03-29T17:52:03.110 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:32 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644064496 in=38b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.111 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.111 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - OpSequencer::finish_op_in_order: op=19, last_started=19, last_unblocked=19, last_completed=18 2022-03-29T17:52:03.111 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,850 [shard 0] osd - client_request(id=19, detail=m=[osd_op(client.4179.0:32 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644064496 in=38b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.111 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4179.0:34 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.111 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4179.0:34 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.112 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4179.0:34 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.112 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - OpSequencer::start_op: op=20, last_started=19, last_unblocked=19, last_completed=19 2022-03-29T17:52:03.112 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.112 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4179.0:34 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=20b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.112 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.113 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.113 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify-ack 2022-03-29T17:52:03.113 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - handling op notify-ack on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.113 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - do_op_notify_ack 2022-03-29T17:52:03.114 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.114 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,851 [shard 0] osd - notify_ack watch_cookie=140032509772432, notify_id=73014444032 2022-03-29T17:52:03.114 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,851 [shard 0] osd - acking notify on watch 140032509772432,client.4179 2022-03-29T17:52:03.114 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,851 [shard 0] osd - notify_ack 2022-03-29T17:52:03.114 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,851 [shard 0] osd - send_completion -- 0 in progress watchers, timedout watchers 0 2022-03-29T17:52:03.115 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,851 [shard 0] osd - send_completion sending notify replies: notify_reply_t{watcher_gid=4179, watcher_cookie=140032509772432, bl=buffer::list(len=0, 2022-03-29T17:52:03.115 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:52:03.115 INFO:tasks.ceph.osd.0.smithi138.stderr:)} 2022-03-29T17:52:03.115 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:34 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=20b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.115 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.116 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - OpSequencer::finish_op_in_order: op=20, last_started=20, last_unblocked=20, last_completed=19 2022-03-29T17:52:03.116 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - client_request(id=20, detail=m=[osd_op(client.4179.0:34 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=20b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - peering_event(id=474, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:03.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - peering_event(id=474, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:03.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:52:03.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 136.540603638s 2022-03-29T17:52:03.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.540603638s (sent 144.548706055s) 2022-03-29T17:52:03.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,852 [shard 0] osd - peering_event(id=474, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:03.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,853 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.120 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,853 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:52:03.120 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,853 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.540603638s ub 144.548706055s int 16.000000000s))): start 2022-03-29T17:52:03.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.540603638s ub 144.548706055s int 16.000000000s))): got map 17 2022-03-29T17:52:03.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.540603638s ub 144.548706055s int 16.000000000s))): pg present 2022-03-29T17:52:03.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.540603638s ub 144.548706055s int 16.000000000s) for pg: 2.3 2022-03-29T17:52:03.121 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 136.540603638s ub 144.548706055s int 16.000000000s) 2022-03-29T17:52:03.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 144.487350464s 2022-03-29T17:52:03.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 144.487350464s 2022-03-29T17:52:03.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.540603638s ub 144.548706055s int 16.000000000s))): submitting ctx 2022-03-29T17:52:03.122 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,854 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,855 [shard 0] osd - peering_event(id=474, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:03.125 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,855 [shard 0] osd - peering_event(id=218, detail=PeeringEvent(from=1 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.540603638s ub 144.548706055s int 16.000000000s))): complete 2022-03-29T17:52:03.125 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,856 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:03.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,856 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:03.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,856 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:52:03.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,856 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 136.488082886s 2022-03-29T17:52:03.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,856 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.488082886s (sent 144.489608765s) 2022-03-29T17:52:03.126 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,856 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.127 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,856 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:03.127 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,856 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.127 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,857 [shard 0] osd - peering_event(id=219, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:03.127 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,866 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:03.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,866 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:03.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,866 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:52:03.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,866 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 136.498077393s 2022-03-29T17:52:03.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,866 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.498077393s (sent 144.499664307s) 2022-03-29T17:52:03.128 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,866 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.129 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,866 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:03.129 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,866 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.129 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,867 [shard 0] osd - peering_event(id=220, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:03.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,872 [shard 0] osd - peering_event(id=475, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:03.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,872 [shard 0] osd - peering_event(id=476, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:03.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,872 [shard 0] osd - peering_event(id=477, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:03.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,872 [shard 0] osd - peering_event(id=478, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:03.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,872 [shard 0] osd - peering_event(id=476, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:03.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,872 [shard 0] osd - peering_event(id=475, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:03.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,872 [shard 0] osd - peering_event(id=478, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:03.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,872 [shard 0] osd - peering_event(id=477, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:03.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:52:03.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 136.563827515s 2022-03-29T17:52:03.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.563827515s (sent 144.568969727s) 2022-03-29T17:52:03.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - peering_event(id=476, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:03.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:52:03.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 136.563491821s 2022-03-29T17:52:03.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.563491821s (sent 144.569213867s) 2022-03-29T17:52:03.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - peering_event(id=475, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:03.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:52:03.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 136.564376831s 2022-03-29T17:52:03.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.564376831s (sent 144.569427490s) 2022-03-29T17:52:03.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - peering_event(id=477, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:03.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:52:03.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 136.564651489s 2022-03-29T17:52:03.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.564651489s (sent 144.569808960s) 2022-03-29T17:52:03.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,873 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,874 [shard 0] osd - peering_event(id=478, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:03.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,874 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,874 [shard 0] osd - peering_event(id=476, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:03.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,874 [shard 0] osd - peering_event(id=477, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:03.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,874 [shard 0] osd - peering_event(id=475, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:03.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,874 [shard 0] osd - peering_event(id=478, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:03.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,882 [shard 0] osd - replicated_request(id=101, detail=RepRequest(from=1 req=osd_repop(client.4179.0:33 3.d e17/14 3:b2b71b89:::rbd_object_map.1053953fed53:head v 17'3, mlcod=17'3) v3)) start 2022-03-29T17:52:03.146 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active append_log log((0'0,17'2], crt=17'2) {17'3 (17'2) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:33 2022-03-29T17:52:01.844531+0000 0 ObjectCleanRegions clean_offsets: [0~18,19~12,35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.147 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 lcod 17'2 mlcod 17'2 active add_log_entry 17'3 (17'2) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:33 2022-03-29T17:52:01.844531+0000 0 ObjectCleanRegions clean_offsets: [0~18,19~12,35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.147 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 lcod 17'2 mlcod 17'2 active append_log approx pg log length = 3 2022-03-29T17:52:03.147 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 lcod 17'2 mlcod 17'2 active append_log transaction_applied = 1 2022-03-29T17:52:03.147 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - handle_peering_op on 2.7 from 1 2022-03-29T17:52:03.148 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,883 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563827515s ub 144.568969727s int 16.000000000s))): start 2022-03-29T17:52:03.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563827515s ub 144.568969727s int 16.000000000s))): got map 17 2022-03-29T17:52:03.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563827515s ub 144.568969727s int 16.000000000s))): pg present 2022-03-29T17:52:03.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - handle_peering_op on 2.5 from 1 2022-03-29T17:52:03.149 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563491821s ub 144.569213867s int 16.000000000s))): start 2022-03-29T17:52:03.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563827515s ub 144.568969727s int 16.000000000s) for pg: 2.7 2022-03-29T17:52:03.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 136.563827515s ub 144.568969727s int 16.000000000s) 2022-03-29T17:52:03.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 144.493499756s 2022-03-29T17:52:03.150 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 144.493499756s 2022-03-29T17:52:03.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563827515s ub 144.568969727s int 16.000000000s))): submitting ctx 2022-03-29T17:52:03.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,884 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563491821s ub 144.569213867s int 16.000000000s))): got map 17 2022-03-29T17:52:03.151 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563491821s ub 144.569213867s int 16.000000000s))): pg present 2022-03-29T17:52:03.152 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563827515s ub 144.568969727s int 16.000000000s))): complete 2022-03-29T17:52:03.152 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563491821s ub 144.569213867s int 16.000000000s) for pg: 2.5 2022-03-29T17:52:03.152 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 136.563491821s ub 144.569213867s int 16.000000000s) 2022-03-29T17:52:03.152 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 144.494522095s 2022-03-29T17:52:03.153 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 144.494522095s 2022-03-29T17:52:03.153 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.153 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563491821s ub 144.569213867s int 16.000000000s))): submitting ctx 2022-03-29T17:52:03.153 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.154 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - handle_peering_op on 2.4 from 1 2022-03-29T17:52:03.154 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,885 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564376831s ub 144.569427490s int 16.000000000s))): start 2022-03-29T17:52:03.154 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564376831s ub 144.569427490s int 16.000000000s))): got map 17 2022-03-29T17:52:03.154 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564376831s ub 144.569427490s int 16.000000000s))): pg present 2022-03-29T17:52:03.154 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564376831s ub 144.569427490s int 16.000000000s) for pg: 2.4 2022-03-29T17:52:03.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 136.564376831s ub 144.569427490s int 16.000000000s) 2022-03-29T17:52:03.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 144.495391846s 2022-03-29T17:52:03.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 144.495391846s 2022-03-29T17:52:03.155 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564376831s ub 144.569427490s int 16.000000000s))): submitting ctx 2022-03-29T17:52:03.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - handle_peering_op on 2.6 from 1 2022-03-29T17:52:03.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,886 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=1 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564651489s ub 144.569808960s int 16.000000000s))): start 2022-03-29T17:52:03.156 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=1 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564651489s ub 144.569808960s int 16.000000000s))): got map 17 2022-03-29T17:52:03.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=1 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564651489s ub 144.569808960s int 16.000000000s))): pg present 2022-03-29T17:52:03.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564651489s ub 144.569808960s int 16.000000000s) for pg: 2.6 2022-03-29T17:52:03.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 136.564651489s ub 144.569808960s int 16.000000000s) 2022-03-29T17:52:03.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 144.496109009s 2022-03-29T17:52:03.157 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 144.496109009s 2022-03-29T17:52:03.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=1 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564651489s ub 144.569808960s int 16.000000000s))): submitting ctx 2022-03-29T17:52:03.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.158 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,887 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.563491821s ub 144.569213867s int 16.000000000s))): complete 2022-03-29T17:52:03.159 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,890 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=1 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564376831s ub 144.569427490s int 16.000000000s))): complete 2022-03-29T17:52:03.159 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,890 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=1 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.1 pg_lease(ru 136.564651489s ub 144.569808960s int 16.000000000s))): complete 2022-03-29T17:52:03.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,894 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:52:03.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,894 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.2 pg_lease_ack(ruub 144.548706055s))): start 2022-03-29T17:52:03.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,894 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.2 pg_lease_ack(ruub 144.548706055s))): got map 17 2022-03-29T17:52:03.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.2 pg_lease_ack(ruub 144.548706055s))): pg present 2022-03-29T17:52:03.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.2 pg_lease_ack(ruub 144.548706055s) for pg: 2.3 2022-03-29T17:52:03.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 144.548706055s 2022-03-29T17:52:03.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.548706055s (sent 144.548706055s) 2022-03-29T17:52:03.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.2 pg_lease_ack(ruub 144.548706055s))): submitting ctx 2022-03-29T17:52:03.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:52:03.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,895 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.488082886s ub 144.489608765s int 16.000000000s))): start 2022-03-29T17:52:03.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.488082886s ub 144.489608765s int 16.000000000s))): got map 17 2022-03-29T17:52:03.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.488082886s ub 144.489608765s int 16.000000000s))): pg present 2022-03-29T17:52:03.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:52:03.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.498077393s ub 144.499664307s int 16.000000000s))): start 2022-03-29T17:52:03.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=2.3 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.2 pg_lease_ack(ruub 144.548706055s))): complete 2022-03-29T17:52:03.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.488082886s ub 144.489608765s int 16.000000000s) for pg: 2.0 2022-03-29T17:52:03.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 136.488082886s ub 144.489608765s int 16.000000000s) 2022-03-29T17:52:03.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 144.592514038s 2022-03-29T17:52:03.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 144.592514038s 2022-03-29T17:52:03.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.488082886s ub 144.489608765s int 16.000000000s))): submitting ctx 2022-03-29T17:52:03.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,896 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.498077393s ub 144.499664307s int 16.000000000s))): got map 17 2022-03-29T17:52:03.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.498077393s ub 144.499664307s int 16.000000000s))): pg present 2022-03-29T17:52:03.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.498077393s ub 144.499664307s int 16.000000000s) for pg: 2.1 2022-03-29T17:52:03.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 136.498077393s ub 144.499664307s int 16.000000000s) 2022-03-29T17:52:03.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 144.593093872s 2022-03-29T17:52:03.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 144.593093872s 2022-03-29T17:52:03.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.498077393s ub 144.499664307s int 16.000000000s))): submitting ctx 2022-03-29T17:52:03.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,897 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.488082886s ub 144.489608765s int 16.000000000s))): complete 2022-03-29T17:52:03.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,898 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.2 pg_lease(ru 136.498077393s ub 144.499664307s int 16.000000000s))): complete 2022-03-29T17:52:03.171 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,900 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:03.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,900 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:03.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,901 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.2 2022-03-29T17:52:03.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,901 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 136.501800537s 2022-03-29T17:52:03.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,901 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 136.501800537s (sent 144.509994507s) 2022-03-29T17:52:03.172 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,901 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,901 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:03.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,901 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.173 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,901 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=0 pgid=2.2 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:03.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,924 [shard 0] osd - handle_peering_op on 2.7 from 0 2022-03-29T17:52:03.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,924 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=0 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.568969727s))): start 2022-03-29T17:52:03.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=0 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.568969727s))): got map 17 2022-03-29T17:52:03.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=0 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.568969727s))): pg present 2022-03-29T17:52:03.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:33 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.assert_locked in=44b, call rbd.object_map_update in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:b2b71b89:::rbd_object_map.1053953fed53:head sending reply 2022-03-29T17:52:03.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.568969727s) for pg: 2.7 2022-03-29T17:52:03.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.568969727s 2022-03-29T17:52:03.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.568969727s (sent 144.568969727s) 2022-03-29T17:52:03.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=0 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.568969727s))): submitting ctx 2022-03-29T17:52:03.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,925 [shard 0] osd - with_head_obc: released 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - OpSequencer::finish_op_in_order: op=153, last_started=153, last_unblocked=153, last_completed=152 2022-03-29T17:52:03.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - client_request(id=153, detail=m=[osd_op(client.4179.0:33 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.assert_locked in=44b, call rbd.object_map_update in=38b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - handle_peering_op on 2.5 from 0 2022-03-29T17:52:03.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=0 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569213867s))): start 2022-03-29T17:52:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=0 pgid=2.7 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.568969727s))): complete 2022-03-29T17:52:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=0 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569213867s))): got map 17 2022-03-29T17:52:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=0 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569213867s))): pg present 2022-03-29T17:52:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569213867s) for pg: 2.5 2022-03-29T17:52:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569213867s 2022-03-29T17:52:03.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569213867s (sent 144.569213867s) 2022-03-29T17:52:03.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=0 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569213867s))): submitting ctx 2022-03-29T17:52:03.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,926 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - handle_peering_op on 2.4 from 0 2022-03-29T17:52:03.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=0 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569427490s))): start 2022-03-29T17:52:03.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=0 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569427490s))): got map 17 2022-03-29T17:52:03.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=0 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569427490s))): pg present 2022-03-29T17:52:03.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=0 pgid=2.5 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569213867s))): complete 2022-03-29T17:52:03.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569427490s) for pg: 2.4 2022-03-29T17:52:03.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569427490s 2022-03-29T17:52:03.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569427490s (sent 144.569427490s) 2022-03-29T17:52:03.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=0 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569427490s))): submitting ctx 2022-03-29T17:52:03.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - handle_peering_op on 2.6 from 0 2022-03-29T17:52:03.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,927 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=0 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569808960s))): start 2022-03-29T17:52:03.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=0 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569808960s))): got map 17 2022-03-29T17:52:03.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=0 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569808960s))): pg present 2022-03-29T17:52:03.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=0 pgid=2.4 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569427490s))): complete 2022-03-29T17:52:03.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - handle_peering_op on 2.2 from 0 2022-03-29T17:52:03.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=0 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.0 pg_lease(ru 136.501800537s ub 144.509994507s int 16.000000000s))): start 2022-03-29T17:52:03.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569808960s) for pg: 2.6 2022-03-29T17:52:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569808960s 2022-03-29T17:52:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569808960s (sent 144.569808960s) 2022-03-29T17:52:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=0 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569808960s))): submitting ctx 2022-03-29T17:52:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=0 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.0 pg_lease(ru 136.501800537s ub 144.509994507s int 16.000000000s))): got map 17 2022-03-29T17:52:03.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,928 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=0 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.0 pg_lease(ru 136.501800537s ub 144.509994507s int 16.000000000s))): pg present 2022-03-29T17:52:03.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,929 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.0 pg_lease(ru 136.501800537s ub 144.509994507s int 16.000000000s) for pg: 2.2 2022-03-29T17:52:03.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,929 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 136.501800537s ub 144.509994507s int 16.000000000s) 2022-03-29T17:52:03.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,929 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 144.625045776s 2022-03-29T17:52:03.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,929 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 144.625045776s 2022-03-29T17:52:03.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,929 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,929 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=0 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.0 pg_lease(ru 136.501800537s ub 144.509994507s int 16.000000000s))): submitting ctx 2022-03-29T17:52:03.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,929 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,929 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=0 pgid=2.6 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.0 pg_lease_ack(ruub 144.569808960s))): complete 2022-03-29T17:52:03.192 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4179.0:35 3.9 3.200f8289 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.193 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4179.0:35 3.9 3.200f8289 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=0 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLease epoch 17 from osd.0 pg_lease(ru 136.501800537s ub 144.509994507s int 16.000000000s))): complete 2022-03-29T17:52:03.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4179.0:35 3.9 3.200f8289 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - OpSequencer::start_op: op=21, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:03.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - do_recover_missing check for recovery, 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head 2022-03-29T17:52:03.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4179.0:35 3.9 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head {set-alloc-hint object_size 4194304 write_size 4194304, write 0~256 in=256b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.196 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - with_head_obc 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head 2022-03-29T17:52:03.197 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,930 [shard 0] osd - with_head_obc: cache miss on 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head 2022-03-29T17:52:03.197 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - load_metadata: object 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head doesn't exist, returning empty metadata 2022-03-29T17:52:03.197 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - load_head_obc: loaded obs 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head 2022-03-29T17:52:03.197 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - load_head_obc: returning obc 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head 2022-03-29T17:52:03.198 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - do_osd_ops_execute: object 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head - handling op set-alloc-hint 2022-03-29T17:52:03.198 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - handling op set-alloc-hint on object 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head 2022-03-29T17:52:03.198 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - do_osd_ops_execute: object 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head - handling op write 2022-03-29T17:52:03.198 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - handling op write on object 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head 2022-03-29T17:52:03.198 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - do_osd_ops_execute: object 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head all operations successful 2022-03-29T17:52:03.199 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - do_osd_ops_execute: object 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head submitting txn 2022-03-29T17:52:03.199 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:03.199 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head by client.4179.0:35 2022-03-29T17:52:01.925949+0000 0 ObjectCleanRegions clean_offsets: [256~18446744073709551359], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.199 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - pg_epoch 17 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 17'1 (0'0) modify 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head by client.4179.0:35 2022-03-29T17:52:01.925949+0000 0 ObjectCleanRegions clean_offsets: [256~18446744073709551359], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.200 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - pg_epoch 17 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:03.200 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - pg_epoch 17 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.200 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.200 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.200 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,931 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,932 [shard 0] osd - replicated_request(id=3, detail=RepRequest(from=0 req=osd_repop(client.4179.0:35 3.9 e17/14 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head v 17'1, mlcod=17'1) v3)) start 2022-03-29T17:52:03.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,933 [shard 0] osd - pg_epoch 17 pg[3.9( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {17'1 (0'0) modify 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head by client.4179.0:35 2022-03-29T17:52:01.925949+0000 0 ObjectCleanRegions clean_offsets: [256~18446744073709551359], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,933 [shard 0] osd - pg_epoch 17 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 17'1 (0'0) modify 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head by client.4179.0:35 2022-03-29T17:52:01.925949+0000 0 ObjectCleanRegions clean_offsets: [256~18446744073709551359], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,933 [shard 0] osd - pg_epoch 17 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:03.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,933 [shard 0] osd - pg_epoch 17 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:03.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,933 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,933 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 17'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,933 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,935 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:52:03.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,935 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.489608765s))): start 2022-03-29T17:52:03.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,935 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.489608765s))): got map 17 2022-03-29T17:52:03.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.489608765s))): pg present 2022-03-29T17:52:03.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.489608765s) for pg: 2.0 2022-03-29T17:52:03.207 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 144.489608765s 2022-03-29T17:52:03.207 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.489608765s (sent 144.489608765s) 2022-03-29T17:52:03.207 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.207 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.489608765s))): submitting ctx 2022-03-29T17:52:03.207 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.208 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:52:03.208 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.499664307s))): start 2022-03-29T17:52:03.208 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.499664307s))): got map 17 2022-03-29T17:52:03.208 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.499664307s))): pg present 2022-03-29T17:52:03.209 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - peering_event(id=221, detail=PeeringEvent(from=1 pgid=2.0 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.489608765s))): complete 2022-03-29T17:52:03.209 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.499664307s) for pg: 2.1 2022-03-29T17:52:03.209 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,936 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 144.499664307s 2022-03-29T17:52:03.209 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,937 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.499664307s (sent 144.499664307s) 2022-03-29T17:52:03.209 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,937 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.210 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,937 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.499664307s))): submitting ctx 2022-03-29T17:52:03.210 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,937 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.210 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:01,937 [shard 0] osd - peering_event(id=222, detail=PeeringEvent(from=1 pgid=2.1 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.499664307s))): complete 2022-03-29T17:52:03.215 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,935 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:35 3.9 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head {set-alloc-hint object_size 4194304 write_size 4194304, write 0~256 in=256b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8 - object 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head sending reply 2022-03-29T17:52:03.216 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,938 [shard 0] osd - with_head_obc: released 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head 2022-03-29T17:52:03.216 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,938 [shard 0] osd - OpSequencer::finish_op_in_order: op=21, last_started=21, last_unblocked=21, last_completed=0 2022-03-29T17:52:03.216 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,938 [shard 0] osd - client_request(id=21, detail=m=[osd_op(client.4179.0:35 3.9 3:9141f004:::rbd_data.1053953fed53.0000000000000001:head {set-alloc-hint object_size 4194304 write_size 4194304, write 0~256 in=256b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.216 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4179.0:36 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4179.0:36 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4179.0:36 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - OpSequencer::start_op: op=22, last_started=20, last_unblocked=20, last_completed=20 2022-03-29T17:52:03.217 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4179.0:36 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644061600 in=46b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify 2022-03-29T17:52:03.218 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - handling op notify on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.219 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - do_op_notify, msg epoch: 17 2022-03-29T17:52:03.219 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,939 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.219 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,939 [shard 0] osd - start_notify adding notify(id=73014444033) 2022-03-29T17:52:03.219 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,939 [shard 0] osd - send_notify_msg for notify(id=73014444033) 2022-03-29T17:52:03.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:36 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644061600 in=46b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - OpSequencer::finish_op_in_order: op=22, last_started=22, last_unblocked=22, last_completed=20 2022-03-29T17:52:03.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - client_request(id=22, detail=m=[osd_op(client.4179.0:36 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644061600 in=46b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.220 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4179.0:37 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): start 2022-03-29T17:52:03.221 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4179.0:37 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): in repeat 2022-03-29T17:52:03.221 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4179.0:37 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e17) v8]) same_interval_since: 14 2022-03-29T17:52:03.221 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - OpSequencer::start_op: op=23, last_started=22, last_unblocked=22, last_completed=22 2022-03-29T17:52:03.221 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.221 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4179.0:37 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): got obc lock 2022-03-29T17:52:03.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,940 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify-ack 2022-03-29T17:52:03.222 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - handling op notify-ack on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - do_op_notify_ack 2022-03-29T17:52:03.223 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.223 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,941 [shard 0] osd - notify_ack watch_cookie=140032509772432, notify_id=73014444033 2022-03-29T17:52:03.223 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,941 [shard 0] osd - acking notify on watch 140032509772432,client.4179 2022-03-29T17:52:03.223 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,941 [shard 0] osd - notify_ack 2022-03-29T17:52:03.224 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:01,941 [shard 0] osd - send_completion -- 0 in progress watchers, timedout watchers 0 2022-03-29T17:52:03.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - send_completion sending notify replies: notify_reply_t{watcher_gid=4179, watcher_cookie=140032509772432, bl=buffer::list(len=10, 2022-03-29T17:52:03.224 INFO:tasks.ceph.osd.0.smithi138.stderr: buffer::ptr(20~10 0x615000222fc9 in raw 0x615000222fb5 len 30 nref 4) 2022-03-29T17:52:03.224 INFO:tasks.ceph.osd.0.smithi138.stderr:)} 2022-03-29T17:52:03.224 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:37 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - OpSequencer::finish_op_in_order: op=23, last_started=23, last_unblocked=23, last_completed=22 2022-03-29T17:52:03.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,941 [shard 0] osd - client_request(id=23, detail=m=[osd_op(client.4179.0:37 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e17) v8]): destroying 2022-03-29T17:52:03.225 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,965 [shard 0] osd - handle_peering_op on 2.2 from 1 2022-03-29T17:52:03.226 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,965 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.509994507s))): start 2022-03-29T17:52:03.226 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,966 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.509994507s))): got map 17 2022-03-29T17:52:03.226 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,966 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.509994507s))): pg present 2022-03-29T17:52:03.226 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,966 [shard 0] osd - do_peering_event handling epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.509994507s) for pg: 2.2 2022-03-29T17:52:03.226 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,966 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 144.509994507s 2022-03-29T17:52:03.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,966 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.509994507s (sent 144.509994507s) 2022-03-29T17:52:03.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,966 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,966 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.509994507s))): submitting ctx 2022-03-29T17:52:03.227 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,966 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.228 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:01,967 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.2 sent=17 requested=17 evt=epoch_sent: 17 epoch_requested: 17 MLeaseAck epoch 17 from osd.1 pg_lease_ack(ruub 144.509994507s))): complete 2022-03-29T17:52:03.228 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:02,304 [shard 0] osd - handle_osd_map osd_map(17..18 src has 1..18) v4 2022-03-29T17:52:03.229 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:02,304 [shard 0] osd - handle_osd_map epochs [17..18], i have 17, src has [1..18] 2022-03-29T17:52:03.229 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,305 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:03.229 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:02,307 [shard 0] osd - osd.1: committed_osd_maps(18, 18) 2022-03-29T17:52:03.229 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:02,307 [shard 0] osd - osd.1: map e 18 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:03.231 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,307 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4179.0:39 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.231 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,307 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4179.0:39 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.231 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,307 [shard 0] osd - evt epoch is 18, i have 17, will wait 2022-03-29T17:52:03.231 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,308 [shard 0] osd - osdmap_subscribe(17) 2022-03-29T17:52:03.231 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,308 [shard 0] osd - handle_osd_map osd_map(18..18 src has 1..18) v4 2022-03-29T17:52:03.232 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,308 [shard 0] osd - handle_osd_map epochs [18..18], i have 17, src has [1..18] 2022-03-29T17:52:03.232 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:03.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=153, detail=PGAdvanceMap(pg=1.0 from=17 to=18)): start 2022-03-29T17:52:03.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=154, detail=PGAdvanceMap(pg=2.0 from=17 to=18)): start 2022-03-29T17:52:03.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=155, detail=PGAdvanceMap(pg=2.1 from=17 to=18)): start 2022-03-29T17:52:03.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=156, detail=PGAdvanceMap(pg=2.2 from=17 to=18)): start 2022-03-29T17:52:03.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=157, detail=PGAdvanceMap(pg=2.3 from=17 to=18)): start 2022-03-29T17:52:03.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=158, detail=PGAdvanceMap(pg=2.4 from=17 to=18)): start 2022-03-29T17:52:03.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=159, detail=PGAdvanceMap(pg=2.5 from=17 to=18)): start 2022-03-29T17:52:03.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=160, detail=PGAdvanceMap(pg=2.6 from=17 to=18)): start 2022-03-29T17:52:03.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=161, detail=PGAdvanceMap(pg=2.7 from=17 to=18)): start 2022-03-29T17:52:03.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=162, detail=PGAdvanceMap(pg=3.0 from=17 to=18)): start 2022-03-29T17:52:03.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=163, detail=PGAdvanceMap(pg=3.2 from=17 to=18)): start 2022-03-29T17:52:03.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,309 [shard 0] osd - pg_advance_map(id=164, detail=PGAdvanceMap(pg=3.4 from=17 to=18)): start 2022-03-29T17:52:03.245 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=165, detail=PGAdvanceMap(pg=3.5 from=17 to=18)): start 2022-03-29T17:52:03.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=166, detail=PGAdvanceMap(pg=3.6 from=17 to=18)): start 2022-03-29T17:52:03.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=167, detail=PGAdvanceMap(pg=3.7 from=17 to=18)): start 2022-03-29T17:52:03.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=168, detail=PGAdvanceMap(pg=3.8 from=17 to=18)): start 2022-03-29T17:52:03.246 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=169, detail=PGAdvanceMap(pg=3.a from=17 to=18)): start 2022-03-29T17:52:03.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=170, detail=PGAdvanceMap(pg=3.b from=17 to=18)): start 2022-03-29T17:52:03.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=171, detail=PGAdvanceMap(pg=3.c from=17 to=18)): start 2022-03-29T17:52:03.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=172, detail=PGAdvanceMap(pg=3.d from=17 to=18)): start 2022-03-29T17:52:03.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=173, detail=PGAdvanceMap(pg=3.f from=17 to=18)): start 2022-03-29T17:52:03.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=174, detail=PGAdvanceMap(pg=3.10 from=17 to=18)): start 2022-03-29T17:52:03.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=175, detail=PGAdvanceMap(pg=3.11 from=17 to=18)): start 2022-03-29T17:52:03.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=176, detail=PGAdvanceMap(pg=3.12 from=17 to=18)): start 2022-03-29T17:52:03.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=177, detail=PGAdvanceMap(pg=3.13 from=17 to=18)): start 2022-03-29T17:52:03.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=178, detail=PGAdvanceMap(pg=3.14 from=17 to=18)): start 2022-03-29T17:52:03.248 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=179, detail=PGAdvanceMap(pg=3.15 from=17 to=18)): start 2022-03-29T17:52:03.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=180, detail=PGAdvanceMap(pg=3.16 from=17 to=18)): start 2022-03-29T17:52:03.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=181, detail=PGAdvanceMap(pg=3.17 from=17 to=18)): start 2022-03-29T17:52:03.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=182, detail=PGAdvanceMap(pg=3.19 from=17 to=18)): start 2022-03-29T17:52:03.249 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=183, detail=PGAdvanceMap(pg=3.1a from=17 to=18)): start 2022-03-29T17:52:03.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=184, detail=PGAdvanceMap(pg=3.1b from=17 to=18)): start 2022-03-29T17:52:03.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=185, detail=PGAdvanceMap(pg=3.1c from=17 to=18)): start 2022-03-29T17:52:03.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=186, detail=PGAdvanceMap(pg=3.1e from=17 to=18)): start 2022-03-29T17:52:03.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_advance_map(id=187, detail=PGAdvanceMap(pg=3.1f from=17 to=18)): start 2022-03-29T17:52:03.250 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,310 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,311 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active state: Started advmap 2022-03-29T17:52:03.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.260 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,312 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:14 2022-03-29T17:52:03.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.261 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.262 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.263 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.264 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 17 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.265 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Active advmap 2022-03-29T17:52:03.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,313 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Started advmap 2022-03-29T17:52:03.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.266 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.267 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.268 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,314 [shard 0] osd - osd.0: committed_osd_maps(18, 18) 2022-03-29T17:52:03.268 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,314 [shard 0] osd - osd.0: map e 18 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:03.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.276 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,314 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.277 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active activate_map 2022-03-29T17:52:03.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.278 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.279 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.281 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,315 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.282 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.283 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.284 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean state: Active advmap 2022-03-29T17:52:03.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean state: Started advmap 2022-03-29T17:52:03.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.285 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.286 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:14 2022-03-29T17:52:03.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,316 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean activate_map 2022-03-29T17:52:03.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean state: handle ActMap primary 2022-03-29T17:52:03.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.291 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=86, detail=PGAdvanceMap(pg=1.0 from=17 to=18)): start 2022-03-29T17:52:03.291 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=87, detail=PGAdvanceMap(pg=2.2 from=17 to=18)): start 2022-03-29T17:52:03.291 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=88, detail=PGAdvanceMap(pg=2.4 from=17 to=18)): start 2022-03-29T17:52:03.291 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=89, detail=PGAdvanceMap(pg=2.5 from=17 to=18)): start 2022-03-29T17:52:03.292 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=90, detail=PGAdvanceMap(pg=2.6 from=17 to=18)): start 2022-03-29T17:52:03.292 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=91, detail=PGAdvanceMap(pg=2.7 from=17 to=18)): start 2022-03-29T17:52:03.292 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=92, detail=PGAdvanceMap(pg=3.1 from=17 to=18)): start 2022-03-29T17:52:03.292 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=93, detail=PGAdvanceMap(pg=3.3 from=17 to=18)): start 2022-03-29T17:52:03.293 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=94, detail=PGAdvanceMap(pg=3.6 from=17 to=18)): start 2022-03-29T17:52:03.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=95, detail=PGAdvanceMap(pg=3.9 from=17 to=18)): start 2022-03-29T17:52:03.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=96, detail=PGAdvanceMap(pg=3.a from=17 to=18)): start 2022-03-29T17:52:03.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=97, detail=PGAdvanceMap(pg=3.b from=17 to=18)): start 2022-03-29T17:52:03.297 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,317 [shard 0] osd - pg_advance_map(id=98, detail=PGAdvanceMap(pg=3.c from=17 to=18)): start 2022-03-29T17:52:03.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=99, detail=PGAdvanceMap(pg=3.d from=17 to=18)): start 2022-03-29T17:52:03.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=100, detail=PGAdvanceMap(pg=3.e from=17 to=18)): start 2022-03-29T17:52:03.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=101, detail=PGAdvanceMap(pg=3.f from=17 to=18)): start 2022-03-29T17:52:03.298 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=102, detail=PGAdvanceMap(pg=3.10 from=17 to=18)): start 2022-03-29T17:52:03.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=103, detail=PGAdvanceMap(pg=3.12 from=17 to=18)): start 2022-03-29T17:52:03.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=104, detail=PGAdvanceMap(pg=3.13 from=17 to=18)): start 2022-03-29T17:52:03.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=105, detail=PGAdvanceMap(pg=3.15 from=17 to=18)): start 2022-03-29T17:52:03.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=106, detail=PGAdvanceMap(pg=3.17 from=17 to=18)): start 2022-03-29T17:52:03.299 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=107, detail=PGAdvanceMap(pg=3.18 from=17 to=18)): start 2022-03-29T17:52:03.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 17 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active state: Started advmap 2022-03-29T17:52:03.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:14 2022-03-29T17:52:03.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.303 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.304 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.305 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=108, detail=PGAdvanceMap(pg=3.1b from=17 to=18)): start 2022-03-29T17:52:03.305 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=109, detail=PGAdvanceMap(pg=3.1c from=17 to=18)): start 2022-03-29T17:52:03.305 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=110, detail=PGAdvanceMap(pg=3.1d from=17 to=18)): start 2022-03-29T17:52:03.305 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_advance_map(id=111, detail=PGAdvanceMap(pg=3.1f from=17 to=18)): start 2022-03-29T17:52:03.306 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,318 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:03.306 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.306 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.306 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.308 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.309 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean activate_map 2022-03-29T17:52:03.310 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.311 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: handle ActMap primary 2022-03-29T17:52:03.312 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.312 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.312 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.312 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.313 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.316 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.318 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.318 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.318 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.318 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 17 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.319 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active state: Started advmap 2022-03-29T17:52:03.319 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.319 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.319 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.320 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.320 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.320 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active activate_map 2022-03-29T17:52:03.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.322 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.322 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.323 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.323 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 17 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,320 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.324 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.325 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.326 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.326 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 17 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.326 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Active advmap 2022-03-29T17:52:03.326 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Started advmap 2022-03-29T17:52:03.327 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.328 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,321 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.330 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:14 2022-03-29T17:52:03.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.331 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.333 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 17 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.333 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.333 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.333 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.334 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.334 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.334 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.334 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.334 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.335 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.335 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.335 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.337 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,322 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.338 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.340 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 17 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.340 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.341 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.341 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.341 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 17 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.341 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.341 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.342 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 17 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.342 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.342 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 17 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Active advmap 2022-03-29T17:52:03.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Started advmap 2022-03-29T17:52:03.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,323 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:14 2022-03-29T17:52:03.348 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 17 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.348 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.348 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.348 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 17 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.349 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.349 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.349 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.349 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.349 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.350 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.350 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 17 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,324 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.353 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.355 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 17 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.356 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.357 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.357 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.357 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.357 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.359 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.360 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,325 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:03.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.364 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.365 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.366 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.366 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.366 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.366 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.366 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.367 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.367 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.369 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active activate_map 2022-03-29T17:52:03.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.370 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,326 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 17 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean state: Active advmap 2022-03-29T17:52:03.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean state: Started advmap 2022-03-29T17:52:03.373 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 17 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:03.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Active advmap 2022-03-29T17:52:03.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Started advmap 2022-03-29T17:52:03.374 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.375 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 17 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean activate_map 2022-03-29T17:52:03.376 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: handle ActMap primary 2022-03-29T17:52:03.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.377 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,327 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.378 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.379 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.380 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.381 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean activate_map 2022-03-29T17:52:03.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.382 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: handle ActMap primary 2022-03-29T17:52:03.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.383 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.384 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.386 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,328 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.387 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active state: Started advmap 2022-03-29T17:52:03.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.389 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 17 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.390 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean activate_map 2022-03-29T17:52:03.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean state: handle ActMap primary 2022-03-29T17:52:03.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.391 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.392 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.393 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 17 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active advmap 2022-03-29T17:52:03.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Started advmap 2022-03-29T17:52:03.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.396 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,329 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.396 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.396 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.396 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:13 2022-03-29T17:52:03.397 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 17 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.398 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.399 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,330 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map 2022-03-29T17:52:03.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: handle ActMap primary 2022-03-29T17:52:03.400 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean publish_stats_to_osd 17:17 2022-03-29T17:52:03.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.401 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.403 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean activate_map 2022-03-29T17:52:03.403 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.403 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean state: handle ActMap primary 2022-03-29T17:52:03.403 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.404 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.404 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.404 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.404 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.406 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.407 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.407 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,331 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.407 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.407 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.408 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.408 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.408 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.409 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.410 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.411 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.411 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.412 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.413 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:03.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:11 2022-03-29T17:52:03.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.416 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,332 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.416 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.416 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:03.417 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.417 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.417 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.417 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.418 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.418 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.418 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.418 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.418 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.419 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.420 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_advance_map(id=171, detail=PGAdvanceMap(pg=3.c from=17 to=18)): complete 2022-03-29T17:52:03.421 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_advance_map(id=166, detail=PGAdvanceMap(pg=3.6 from=17 to=18)): complete 2022-03-29T17:52:03.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.422 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.423 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.424 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_advance_map(id=163, detail=PGAdvanceMap(pg=3.2 from=17 to=18)): complete 2022-03-29T17:52:03.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:03.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.425 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_advance_map(id=176, detail=PGAdvanceMap(pg=3.12 from=17 to=18)): complete 2022-03-29T17:52:03.426 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_advance_map(id=164, detail=PGAdvanceMap(pg=3.4 from=17 to=18)): complete 2022-03-29T17:52:03.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=154, detail=PGAdvanceMap(pg=2.0 from=17 to=18)): complete 2022-03-29T17:52:03.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=165, detail=PGAdvanceMap(pg=3.5 from=17 to=18)): complete 2022-03-29T17:52:03.427 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=157, detail=PGAdvanceMap(pg=2.3 from=17 to=18)): complete 2022-03-29T17:52:03.428 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,334 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.428 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.429 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.429 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.429 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.429 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.430 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=169, detail=PGAdvanceMap(pg=3.a from=17 to=18)): complete 2022-03-29T17:52:03.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=180, detail=PGAdvanceMap(pg=3.16 from=17 to=18)): complete 2022-03-29T17:52:03.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=179, detail=PGAdvanceMap(pg=3.15 from=17 to=18)): complete 2022-03-29T17:52:03.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=159, detail=PGAdvanceMap(pg=2.5 from=17 to=18)): complete 2022-03-29T17:52:03.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=174, detail=PGAdvanceMap(pg=3.10 from=17 to=18)): complete 2022-03-29T17:52:03.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_advance_map(id=158, detail=PGAdvanceMap(pg=2.4 from=17 to=18)): complete 2022-03-29T17:52:03.434 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 17 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {0, 1}/{0, 1} -- 0/0 2022-03-29T17:52:03.434 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.434 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.434 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 17 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,335 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 17 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.435 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active state: Started advmap 2022-03-29T17:52:03.436 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.436 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.436 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.436 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.436 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=177, detail=PGAdvanceMap(pg=3.13 from=17 to=18)): complete 2022-03-29T17:52:03.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=178, detail=PGAdvanceMap(pg=3.14 from=17 to=18)): complete 2022-03-29T17:52:03.438 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=181, detail=PGAdvanceMap(pg=3.17 from=17 to=18)): complete 2022-03-29T17:52:03.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=167, detail=PGAdvanceMap(pg=3.7 from=17 to=18)): complete 2022-03-29T17:52:03.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=161, detail=PGAdvanceMap(pg=2.7 from=17 to=18)): complete 2022-03-29T17:52:03.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=168, detail=PGAdvanceMap(pg=3.8 from=17 to=18)): complete 2022-03-29T17:52:03.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=185, detail=PGAdvanceMap(pg=3.1c from=17 to=18)): complete 2022-03-29T17:52:03.439 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=183, detail=PGAdvanceMap(pg=3.1a from=17 to=18)): complete 2022-03-29T17:52:03.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=173, detail=PGAdvanceMap(pg=3.f from=17 to=18)): complete 2022-03-29T17:52:03.440 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_advance_map(id=162, detail=PGAdvanceMap(pg=3.0 from=17 to=18)): complete 2022-03-29T17:52:03.441 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.441 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,336 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.442 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.443 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_advance_map(id=160, detail=PGAdvanceMap(pg=2.6 from=17 to=18)): complete 2022-03-29T17:52:03.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_advance_map(id=187, detail=PGAdvanceMap(pg=3.1f from=17 to=18)): complete 2022-03-29T17:52:03.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_advance_map(id=175, detail=PGAdvanceMap(pg=3.11 from=17 to=18)): complete 2022-03-29T17:52:03.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_advance_map(id=172, detail=PGAdvanceMap(pg=3.d from=17 to=18)): complete 2022-03-29T17:52:03.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_advance_map(id=186, detail=PGAdvanceMap(pg=3.1e from=17 to=18)): complete 2022-03-29T17:52:03.445 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_advance_map(id=170, detail=PGAdvanceMap(pg=3.b from=17 to=18)): complete 2022-03-29T17:52:03.447 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.447 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_epoch 17 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 0}/{1, 0} -- 1/1 2022-03-29T17:52:03.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.448 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,337 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.449 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active activate_map 2022-03-29T17:52:03.450 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.450 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.450 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.450 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map 2022-03-29T17:52:03.451 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=0 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=1 lpr=8 luod=0'0 lua=0'0 crt=12'97 mlcod 12'97 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.451 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.451 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.451 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.451 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.452 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_advance_map(id=184, detail=PGAdvanceMap(pg=3.1b from=17 to=18)): complete 2022-03-29T17:52:03.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_advance_map(id=153, detail=PGAdvanceMap(pg=1.0 from=17 to=18)): complete 2022-03-29T17:52:03.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_advance_map(id=155, detail=PGAdvanceMap(pg=2.1 from=17 to=18)): complete 2022-03-29T17:52:03.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_advance_map(id=182, detail=PGAdvanceMap(pg=3.19 from=17 to=18)): complete 2022-03-29T17:52:03.453 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - pg_advance_map(id=156, detail=PGAdvanceMap(pg=2.2 from=17 to=18)): complete 2022-03-29T17:52:03.454 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:02,338 [shard 0] osd - osd.1: now active 2022-03-29T17:52:03.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,338 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.460 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.461 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.462 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.462 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.462 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - pg_advance_map(id=104, detail=PGAdvanceMap(pg=3.13 from=17 to=18)): complete 2022-03-29T17:52:03.462 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - pg_advance_map(id=99, detail=PGAdvanceMap(pg=3.d from=17 to=18)): complete 2022-03-29T17:52:03.463 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - pg_advance_map(id=101, detail=PGAdvanceMap(pg=3.f from=17 to=18)): complete 2022-03-29T17:52:03.463 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,339 [shard 0] osd - pg_advance_map(id=93, detail=PGAdvanceMap(pg=3.3 from=17 to=18)): complete 2022-03-29T17:52:03.463 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=91, detail=PGAdvanceMap(pg=2.7 from=17 to=18)): complete 2022-03-29T17:52:03.463 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=111, detail=PGAdvanceMap(pg=3.1f from=17 to=18)): complete 2022-03-29T17:52:03.463 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=87, detail=PGAdvanceMap(pg=2.2 from=17 to=18)): complete 2022-03-29T17:52:03.464 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=95, detail=PGAdvanceMap(pg=3.9 from=17 to=18)): complete 2022-03-29T17:52:03.464 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=105, detail=PGAdvanceMap(pg=3.15 from=17 to=18)): complete 2022-03-29T17:52:03.464 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=107, detail=PGAdvanceMap(pg=3.18 from=17 to=18)): complete 2022-03-29T17:52:03.464 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=106, detail=PGAdvanceMap(pg=3.17 from=17 to=18)): complete 2022-03-29T17:52:03.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=102, detail=PGAdvanceMap(pg=3.10 from=17 to=18)): complete 2022-03-29T17:52:03.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=96, detail=PGAdvanceMap(pg=3.a from=17 to=18)): complete 2022-03-29T17:52:03.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=89, detail=PGAdvanceMap(pg=2.5 from=17 to=18)): complete 2022-03-29T17:52:03.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=97, detail=PGAdvanceMap(pg=3.b from=17 to=18)): complete 2022-03-29T17:52:03.465 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=92, detail=PGAdvanceMap(pg=3.1 from=17 to=18)): complete 2022-03-29T17:52:03.466 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=108, detail=PGAdvanceMap(pg=3.1b from=17 to=18)): complete 2022-03-29T17:52:03.466 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=94, detail=PGAdvanceMap(pg=3.6 from=17 to=18)): complete 2022-03-29T17:52:03.466 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=90, detail=PGAdvanceMap(pg=2.6 from=17 to=18)): complete 2022-03-29T17:52:03.466 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=110, detail=PGAdvanceMap(pg=3.1d from=17 to=18)): complete 2022-03-29T17:52:03.466 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=88, detail=PGAdvanceMap(pg=2.4 from=17 to=18)): complete 2022-03-29T17:52:03.467 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=98, detail=PGAdvanceMap(pg=3.c from=17 to=18)): complete 2022-03-29T17:52:03.467 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=103, detail=PGAdvanceMap(pg=3.12 from=17 to=18)): complete 2022-03-29T17:52:03.467 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=109, detail=PGAdvanceMap(pg=3.1c from=17 to=18)): complete 2022-03-29T17:52:03.467 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=86, detail=PGAdvanceMap(pg=1.0 from=17 to=18)): complete 2022-03-29T17:52:03.468 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - pg_advance_map(id=100, detail=PGAdvanceMap(pg=3.e from=17 to=18)): complete 2022-03-29T17:52:03.468 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,340 [shard 0] osd - osd.0: now active 2022-03-29T17:52:03.468 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,340 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4179.0:39 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.468 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,341 [shard 0] osd - OpSequencer::start_op: op=24, last_started=23, last_unblocked=23, last_completed=23 2022-03-29T17:52:03.468 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,341 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,341 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4179.0:39 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_add in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,341 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,341 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,341 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.469 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,341 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.470 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,341 [shard 0] osd - calling method rbd.snapshot_add, num_read=0, num_write=0 2022-03-29T17:52:03.470 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,342 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.470 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,342 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.470 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,342 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,342 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,342 [shard 0] osd - handling op omap-get-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - handling op omap-set-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.471 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - handling op omap-set-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.472 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=6, num_write=2 2022-03-29T17:52:03.472 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.472 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head submitting txn 2022-03-29T17:52:03.472 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean calc_trim_to_aggressive limit = 17'3 2022-03-29T17:52:03.472 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean append_log log((0'0,17'3], crt=17'3) {18'4 (17'3) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:39 2022-03-29T17:52:02.307479+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.473 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'3 lua=0'0 crt=17'3 lcod 17'3 mlcod 17'3 active+clean add_log_entry 18'4 (17'3) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:39 2022-03-29T17:52:02.307479+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.473 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'3 lua=0'0 crt=18'4 lcod 17'3 mlcod 17'3 active+clean append_log approx pg log length = 4 2022-03-29T17:52:03.473 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=17'3 lua=0'0 crt=18'4 lcod 17'3 mlcod 17'3 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.473 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'4, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.474 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,343 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,344 [shard 0] osd - replicated_request(id=5, detail=RepRequest(from=0 req=osd_repop(client.4179.0:39 3.15 e18/14 3:a9e8038f:::rbd_header.1053953fed53:head v 18'4, mlcod=18'4) v3)) start 2022-03-29T17:52:03.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,345 [shard 0] osd - pg_epoch 18 pg[3.15( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active append_log log((0'0,17'3], crt=17'3) {18'4 (17'3) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:39 2022-03-29T17:52:02.307479+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,345 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 lcod 17'3 mlcod 17'3 active add_log_entry 18'4 (17'3) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:39 2022-03-29T17:52:02.307479+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,345 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'4 lcod 17'3 mlcod 17'3 active append_log approx pg log length = 4 2022-03-29T17:52:03.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,345 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'4 lcod 17'3 mlcod 17'3 active append_log transaction_applied = 1 2022-03-29T17:52:03.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,345 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,345 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'4, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,345 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.477 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,385 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:39 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_add in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.478 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,386 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.478 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,386 [shard 0] osd - OpSequencer::finish_op_in_order: op=24, last_started=24, last_unblocked=24, last_completed=23 2022-03-29T17:52:03.478 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,386 [shard 0] osd - client_request(id=24, detail=m=[osd_op(client.4179.0:39 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_add in=42b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - client_request(id=154, detail=m=[osd_op(client.4179.0:40 3.d 3.91d8ed4d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - client_request(id=154, detail=m=[osd_op(client.4179.0:40 3.d 3.91d8ed4d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - client_request(id=154, detail=m=[osd_op(client.4179.0:40 3.d 3.91d8ed4d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - OpSequencer::start_op: op=154, last_started=153, last_unblocked=153, last_completed=153 2022-03-29T17:52:03.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - do_recover_missing check for recovery, 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - client_request(id=154, detail=m=[osd_op(client.4179.0:40 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {read 0~0} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - with_head_obc 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - with_head_obc: found 3:b2b71b89:::rbd_object_map.1053953fed53:head in cache 2022-03-29T17:52:03.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head - handling op read 2022-03-29T17:52:03.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - handling op read on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - read: data length: 35 2022-03-29T17:52:03.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head all operations successful 2022-03-29T17:52:03.483 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,387 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:40 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {read 0~0 out=35b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:b2b71b89:::rbd_object_map.1053953fed53:head sending reply 2022-03-29T17:52:03.483 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,388 [shard 0] osd - with_head_obc: released 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.483 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,388 [shard 0] osd - OpSequencer::finish_op_in_order: op=154, last_started=154, last_unblocked=154, last_completed=153 2022-03-29T17:52:03.483 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,388 [shard 0] osd - client_request(id=154, detail=m=[osd_op(client.4179.0:40 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {read 0~0 out=35b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.509 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,388 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> unknown.? -@41200] ProtocolV2::start_accept(): target_addr=v2:172.21.15.138:41200/0 2022-03-29T17:52:03.509 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,389 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.? -@41200] UPDATE: peer_type=client, policy(lossy=true server=true standby=false resetcheck=false) 2022-03-29T17:52:03.510 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,389 [shard 0] monc - added challenge on [osd.2(client) v2:172.21.15.138:6803/35579 >> client.? -@41200] 2022-03-29T17:52:03.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,390 [shard 0] osd - handle_authentication client.admin has caps allow * 2022-03-29T17:52:03.510 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,391 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4179 172.21.15.138:0/2702680834@41200] established: gs=0, pgs=5, cs=0, client_cookie=0, server_cookie=0, in_seq=0, out_seq=0, out_q=0 2022-03-29T17:52:03.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,391 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:41 3.7 3.5941b7e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,391 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:41 3.7 3.5941b7e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.511 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,391 [shard 0] osd - evt epoch is 18, i have 17, will wait 2022-03-29T17:52:03.511 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,391 [shard 0] osd - osdmap_subscribe(17) 2022-03-29T17:52:03.511 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,392 [shard 0] osd - handle_osd_map osd_map(17..18 src has 1..18) v4 2022-03-29T17:52:03.511 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,392 [shard 0] osd - handle_osd_map epochs [17..18], i have 17, src has [1..18] 2022-03-29T17:52:03.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,392 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:03.512 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,394 [shard 0] osd - osd.2: committed_osd_maps(18, 18) 2022-03-29T17:52:03.512 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,394 [shard 0] osd - osd.2: map e 18 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:03.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=63, detail=PGAdvanceMap(pg=2.0 from=17 to=18)): start 2022-03-29T17:52:03.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=64, detail=PGAdvanceMap(pg=2.1 from=17 to=18)): start 2022-03-29T17:52:03.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=65, detail=PGAdvanceMap(pg=2.3 from=17 to=18)): start 2022-03-29T17:52:03.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=66, detail=PGAdvanceMap(pg=3.0 from=17 to=18)): start 2022-03-29T17:52:03.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=67, detail=PGAdvanceMap(pg=3.1 from=17 to=18)): start 2022-03-29T17:52:03.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=68, detail=PGAdvanceMap(pg=3.2 from=17 to=18)): start 2022-03-29T17:52:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=69, detail=PGAdvanceMap(pg=3.3 from=17 to=18)): start 2022-03-29T17:52:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=70, detail=PGAdvanceMap(pg=3.4 from=17 to=18)): start 2022-03-29T17:52:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=71, detail=PGAdvanceMap(pg=3.5 from=17 to=18)): start 2022-03-29T17:52:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=72, detail=PGAdvanceMap(pg=3.7 from=17 to=18)): start 2022-03-29T17:52:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=73, detail=PGAdvanceMap(pg=3.8 from=17 to=18)): start 2022-03-29T17:52:03.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=74, detail=PGAdvanceMap(pg=3.9 from=17 to=18)): start 2022-03-29T17:52:03.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=75, detail=PGAdvanceMap(pg=3.e from=17 to=18)): start 2022-03-29T17:52:03.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=76, detail=PGAdvanceMap(pg=3.11 from=17 to=18)): start 2022-03-29T17:52:03.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=77, detail=PGAdvanceMap(pg=3.14 from=17 to=18)): start 2022-03-29T17:52:03.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=78, detail=PGAdvanceMap(pg=3.16 from=17 to=18)): start 2022-03-29T17:52:03.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=79, detail=PGAdvanceMap(pg=3.18 from=17 to=18)): start 2022-03-29T17:52:03.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=80, detail=PGAdvanceMap(pg=3.19 from=17 to=18)): start 2022-03-29T17:52:03.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=81, detail=PGAdvanceMap(pg=3.1a from=17 to=18)): start 2022-03-29T17:52:03.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=82, detail=PGAdvanceMap(pg=3.1d from=17 to=18)): start 2022-03-29T17:52:03.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,394 [shard 0] osd - pg_advance_map(id=83, detail=PGAdvanceMap(pg=3.1e from=17 to=18)): start 2022-03-29T17:52:03.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 17 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 17 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 17 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:03.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 17 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active state: Started advmap 2022-03-29T17:52:03.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 17 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,395 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 17 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:03.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 17 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:03.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 17 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 17 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 17 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 17 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 17 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:03.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active state: Started advmap 2022-03-29T17:52:03.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,396 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 17 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 17 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active activate_map 2022-03-29T17:52:03.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 17 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {0, 2}/{0, 2} -- 0/0 2022-03-29T17:52:03.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,397 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 17 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 17 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 17 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:03.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:03.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:03.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,398 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 17 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:13 2022-03-29T17:52:03.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,399 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 17 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:03.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active activate_map 2022-03-29T17:52:03.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active activate_map: Not dirtying info: last_persisted is 17 while current is 18 2022-03-29T17:52:03.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:03.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,400 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 17 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 0}/{2, 0} -- 2/2 2022-03-29T17:52:03.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:03.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:03.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:03.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:10 2022-03-29T17:52:03.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 18 2022-03-29T17:52:03.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,410 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:03.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:03.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:03.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:03.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 17:13 2022-03-29T17:52:03.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 18 2022-03-29T17:52:03.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - log is not dirty 2022-03-29T17:52:03.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:03.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=71, detail=PGAdvanceMap(pg=3.5 from=17 to=18)): complete 2022-03-29T17:52:03.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=72, detail=PGAdvanceMap(pg=3.7 from=17 to=18)): complete 2022-03-29T17:52:03.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=83, detail=PGAdvanceMap(pg=3.1e from=17 to=18)): complete 2022-03-29T17:52:03.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=78, detail=PGAdvanceMap(pg=3.16 from=17 to=18)): complete 2022-03-29T17:52:03.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=66, detail=PGAdvanceMap(pg=3.0 from=17 to=18)): complete 2022-03-29T17:52:03.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=73, detail=PGAdvanceMap(pg=3.8 from=17 to=18)): complete 2022-03-29T17:52:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=63, detail=PGAdvanceMap(pg=2.0 from=17 to=18)): complete 2022-03-29T17:52:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=76, detail=PGAdvanceMap(pg=3.11 from=17 to=18)): complete 2022-03-29T17:52:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,411 [shard 0] osd - pg_advance_map(id=69, detail=PGAdvanceMap(pg=3.3 from=17 to=18)): complete 2022-03-29T17:52:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=67, detail=PGAdvanceMap(pg=3.1 from=17 to=18)): complete 2022-03-29T17:52:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=77, detail=PGAdvanceMap(pg=3.14 from=17 to=18)): complete 2022-03-29T17:52:03.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=81, detail=PGAdvanceMap(pg=3.1a from=17 to=18)): complete 2022-03-29T17:52:03.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=68, detail=PGAdvanceMap(pg=3.2 from=17 to=18)): complete 2022-03-29T17:52:03.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=80, detail=PGAdvanceMap(pg=3.19 from=17 to=18)): complete 2022-03-29T17:52:03.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=74, detail=PGAdvanceMap(pg=3.9 from=17 to=18)): complete 2022-03-29T17:52:03.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=82, detail=PGAdvanceMap(pg=3.1d from=17 to=18)): complete 2022-03-29T17:52:03.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=79, detail=PGAdvanceMap(pg=3.18 from=17 to=18)): complete 2022-03-29T17:52:03.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=75, detail=PGAdvanceMap(pg=3.e from=17 to=18)): complete 2022-03-29T17:52:03.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=65, detail=PGAdvanceMap(pg=2.3 from=17 to=18)): complete 2022-03-29T17:52:03.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=70, detail=PGAdvanceMap(pg=3.4 from=17 to=18)): complete 2022-03-29T17:52:03.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - pg_advance_map(id=64, detail=PGAdvanceMap(pg=2.1 from=17 to=18)): complete 2022-03-29T17:52:03.567 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,412 [shard 0] osd - osd.2: now active 2022-03-29T17:52:03.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:41 3.7 3.5941b7e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - OpSequencer::start_op: op=3, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:03.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - do_recover_missing check for recovery, 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:41 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {writefull 0~35 in=35b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,412 [shard 0] osd - with_head_obc 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - with_head_obc: cache miss on 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - load_metadata: object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head doesn't exist, returning empty metadata 2022-03-29T17:52:03.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - load_head_obc: loaded obs 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - load_head_obc: returning obc 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - do_osd_ops_execute: object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head - handling op writefull 2022-03-29T17:52:03.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - handling op writefull on object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - do_osd_ops_execute: object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head all operations successful 2022-03-29T17:52:03.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - do_osd_ops_execute: object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head submitting txn 2022-03-29T17:52:03.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:03.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {18'1 (0'0) modify 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head by client.4179.0:41 2022-03-29T17:52:02.388366+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 18'1 (0'0) modify 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head by client.4179.0:41 2022-03-29T17:52:02.388366+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:03.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,413 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,414 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,414 [shard 0] osd - replicated_request(id=6, detail=RepRequest(from=2 req=osd_repop(client.4179.0:41 3.7 e18/14 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head v 18'1, mlcod=18'1) v3)) start 2022-03-29T17:52:03.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,415 [shard 0] osd - pg_epoch 18 pg[3.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {18'1 (0'0) modify 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head by client.4179.0:41 2022-03-29T17:52:02.388366+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,415 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 18'1 (0'0) modify 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head by client.4179.0:41 2022-03-29T17:52:02.388366+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,415 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:03.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,415 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:03.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,415 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,415 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,415 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,456 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:41 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {writefull 0~35 in=35b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head sending reply 2022-03-29T17:52:03.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,457 [shard 0] osd - with_head_obc: released 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,457 [shard 0] osd - OpSequencer::finish_op_in_order: op=3, last_started=3, last_unblocked=3, last_completed=0 2022-03-29T17:52:03.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,457 [shard 0] osd - client_request(id=155, detail=m=[osd_op(client.4179.0:42 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,457 [shard 0] osd - client_request(id=155, detail=m=[osd_op(client.4179.0:42 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,457 [shard 0] osd - client_request(id=155, detail=m=[osd_op(client.4179.0:42 3.d 3.91d8ed4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,457 [shard 0] osd - OpSequencer::start_op: op=155, last_started=154, last_unblocked=154, last_completed=154 2022-03-29T17:52:03.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - do_recover_missing check for recovery, 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - client_request(id=155, detail=m=[osd_op(client.4179.0:42 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.assert_locked in=44b, call rbd.object_map_snap_add in=22b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - with_head_obc 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - with_head_obc: found 3:b2b71b89:::rbd_object_map.1053953fed53:head in cache 2022-03-29T17:52:03.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head - handling op call 2022-03-29T17:52:03.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - handling op call on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - calling method lock.assert_locked, num_read=0, num_write=0 2022-03-29T17:52:03.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - handling op getxattr on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.583 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,458 [shard 0] osd - getxattr on obj=3:b2b71b89:::rbd_object_map.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=0 2022-03-29T17:52:03.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head - handling op call 2022-03-29T17:52:03.584 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - handling op call on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - calling method rbd.object_map_snap_add, num_read=1, num_write=0 2022-03-29T17:52:03.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - handling op stat on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - stat os.oi.size=35, os.oi.mtime=1648576321.844531 2022-03-29T17:52:03.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - handling op sync_read on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - read: data length: 35 2022-03-29T17:52:03.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,459 [shard 0] osd - handling op writefull on object 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=1 2022-03-29T17:52:03.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head all operations successful 2022-03-29T17:52:03.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - do_osd_ops_execute: object 3:b2b71b89:::rbd_object_map.1053953fed53:head submitting txn 2022-03-29T17:52:03.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean calc_trim_to_aggressive limit = 17'3 2022-03-29T17:52:03.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'3 mlcod 17'3 active+clean append_log log((0'0,17'3], crt=17'3) {18'4 (17'3) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:42 2022-03-29T17:52:02.457313+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'3 lua=0'0 crt=17'3 lcod 17'3 mlcod 17'3 active+clean add_log_entry 18'4 (17'3) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:42 2022-03-29T17:52:02.457313+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'3 lua=0'0 crt=18'4 lcod 17'3 mlcod 17'3 active+clean append_log approx pg log length = 4 2022-03-29T17:52:03.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'3 lua=0'0 crt=18'4 lcod 17'3 mlcod 17'3 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'4, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,460 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,461 [shard 0] osd - replicated_request(id=102, detail=RepRequest(from=1 req=osd_repop(client.4179.0:42 3.d e18/14 3:b2b71b89:::rbd_object_map.1053953fed53:head v 18'4, mlcod=18'4) v3)) start 2022-03-29T17:52:03.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,462 [shard 0] osd - pg_epoch 18 pg[3.d( v 17'3 (0'0,17'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 mlcod 17'3 active append_log log((0'0,17'3], crt=17'3) {18'4 (17'3) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:42 2022-03-29T17:52:02.457313+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.590 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,462 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'3 lcod 17'3 mlcod 17'3 active add_log_entry 18'4 (17'3) modify 3:b2b71b89:::rbd_object_map.1053953fed53:head by client.4179.0:42 2022-03-29T17:52:02.457313+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,462 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'4 lcod 17'3 mlcod 17'3 active append_log approx pg log length = 4 2022-03-29T17:52:03.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,462 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'4 lcod 17'3 mlcod 17'3 active append_log transaction_applied = 1 2022-03-29T17:52:03.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,462 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,462 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'4, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.591 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,462 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,464 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:42 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.assert_locked in=44b, call rbd.object_map_snap_add in=22b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:b2b71b89:::rbd_object_map.1053953fed53:head sending reply 2022-03-29T17:52:03.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,464 [shard 0] osd - with_head_obc: released 3:b2b71b89:::rbd_object_map.1053953fed53:head 2022-03-29T17:52:03.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,464 [shard 0] osd - OpSequencer::finish_op_in_order: op=155, last_started=155, last_unblocked=155, last_completed=154 2022-03-29T17:52:03.593 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,464 [shard 0] osd - client_request(id=155, detail=m=[osd_op(client.4179.0:42 3.d 3:b2b71b89:::rbd_object_map.1053953fed53:head {call lock.assert_locked in=44b, call rbd.object_map_snap_add in=22b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4179.0:43 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.596 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4179.0:43 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4179.0:43 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - OpSequencer::start_op: op=25, last_started=24, last_unblocked=24, last_completed=24 2022-03-29T17:52:03.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4179.0:43 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032509773648 in=46b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.597 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify 2022-03-29T17:52:03.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,465 [shard 0] osd - handling op notify on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.598 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,466 [shard 0] osd - do_op_notify, msg epoch: 18 2022-03-29T17:52:03.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,466 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.599 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,466 [shard 0] osd - start_notify adding notify(id=77309411330) 2022-03-29T17:52:03.599 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,466 [shard 0] osd - send_notify_msg for notify(id=77309411330) 2022-03-29T17:52:03.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,466 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:43 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032509773648 in=46b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.599 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,466 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,466 [shard 0] osd - OpSequencer::finish_op_in_order: op=25, last_started=25, last_unblocked=25, last_completed=24 2022-03-29T17:52:03.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,466 [shard 0] osd - client_request(id=25, detail=m=[osd_op(client.4179.0:43 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032509773648 in=46b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,466 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4179.0:44 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.600 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,466 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4179.0:44 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4179.0:44 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - OpSequencer::start_op: op=26, last_started=25, last_unblocked=25, last_completed=25 2022-03-29T17:52:03.601 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - client_request(id=3, detail=m=[osd_op(client.4179.0:41 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {writefull 0~35 in=35b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.618 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4179.0:44 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=20b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify-ack 2022-03-29T17:52:03.619 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - handling op notify-ack on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - do_op_notify_ack 2022-03-29T17:52:03.620 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.620 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,467 [shard 0] osd - notify_ack watch_cookie=140032509772432, notify_id=77309411330 2022-03-29T17:52:03.620 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,467 [shard 0] osd - acking notify on watch 140032509772432,client.4179 2022-03-29T17:52:03.620 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,467 [shard 0] osd - notify_ack 2022-03-29T17:52:03.621 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,467 [shard 0] osd - send_completion -- 0 in progress watchers, timedout watchers 0 2022-03-29T17:52:03.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,467 [shard 0] osd - send_completion sending notify replies: notify_reply_t{watcher_gid=4179, watcher_cookie=140032509772432, bl=buffer::list(len=0, 2022-03-29T17:52:03.621 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:52:03.621 INFO:tasks.ceph.osd.0.smithi138.stderr:)} 2022-03-29T17:52:03.621 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,468 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:44 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=20b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,468 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,468 [shard 0] osd - OpSequencer::finish_op_in_order: op=26, last_started=26, last_unblocked=26, last_completed=25 2022-03-29T17:52:03.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,468 [shard 0] osd - client_request(id=26, detail=m=[osd_op(client.4179.0:44 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=20b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.622 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,507 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4179.0:45 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,507 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4179.0:45 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,507 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4179.0:45 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,507 [shard 0] osd - OpSequencer::start_op: op=27, last_started=26, last_unblocked=26, last_completed=26 2022-03-29T17:52:03.623 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,507 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4179.0:45 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644056448 in=22b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.624 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify 2022-03-29T17:52:03.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - handling op notify on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - do_op_notify, msg epoch: 18 2022-03-29T17:52:03.625 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.626 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,508 [shard 0] osd - start_notify adding notify(id=77309411331) 2022-03-29T17:52:03.626 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,508 [shard 0] osd - send_notify_msg for notify(id=77309411331) 2022-03-29T17:52:03.626 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:45 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644056448 in=22b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.627 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.627 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - OpSequencer::finish_op_in_order: op=27, last_started=27, last_unblocked=27, last_completed=26 2022-03-29T17:52:03.627 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,508 [shard 0] osd - client_request(id=27, detail=m=[osd_op(client.4179.0:45 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644056448 in=22b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.628 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4179.0:46 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.628 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4179.0:46 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.628 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4179.0:46 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - OpSequencer::start_op: op=28, last_started=27, last_unblocked=27, last_completed=27 2022-03-29T17:52:03.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.629 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4179.0:46 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.630 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify-ack 2022-03-29T17:52:03.631 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - handling op notify-ack on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.631 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - do_op_notify_ack 2022-03-29T17:52:03.631 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.631 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,509 [shard 0] osd - notify_ack watch_cookie=140032509772432, notify_id=77309411331 2022-03-29T17:52:03.631 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,509 [shard 0] osd - acking notify on watch 140032509772432,client.4179 2022-03-29T17:52:03.632 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,509 [shard 0] osd - notify_ack 2022-03-29T17:52:03.632 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,509 [shard 0] osd - send_completion -- 0 in progress watchers, timedout watchers 0 2022-03-29T17:52:03.632 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - send_completion sending notify replies: notify_reply_t{watcher_gid=4179, watcher_cookie=140032509772432, bl=buffer::list(len=10, 2022-03-29T17:52:03.632 INFO:tasks.ceph.osd.0.smithi138.stderr: buffer::ptr(20~10 0x615000224dc9 in raw 0x615000224db5 len 30 nref 4) 2022-03-29T17:52:03.633 INFO:tasks.ceph.osd.0.smithi138.stderr:)} 2022-03-29T17:52:03.633 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,509 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:46 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.633 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,510 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.633 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,510 [shard 0] osd - OpSequencer::finish_op_in_order: op=28, last_started=28, last_unblocked=28, last_completed=27 2022-03-29T17:52:03.633 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,510 [shard 0] osd - client_request(id=28, detail=m=[osd_op(client.4179.0:46 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.634 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,549 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4179.0:47 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.634 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,549 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4179.0:47 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.634 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,549 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4179.0:47 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.634 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,549 [shard 0] osd - OpSequencer::start_op: op=29, last_started=28, last_unblocked=28, last_completed=28 2022-03-29T17:52:03.635 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,550 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.635 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,550 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4179.0:47 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b, call rbd.get_features in=24b, call rbd.get_flags in=20b, call rbd.get_snapcontext in=18b, call lock.get_info in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.635 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,550 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.635 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,550 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.635 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,550 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,550 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,550 [shard 0] osd - calling method rbd.get_size, num_read=0, num_write=0 2022-03-29T17:52:03.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,550 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,551 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.636 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,551 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=9 while num_read=2, num_write=0 2022-03-29T17:52:03.637 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,551 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.637 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,551 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.637 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,551 [shard 0] osd - calling method rbd.get_features, num_read=2, num_write=0 2022-03-29T17:52:03.637 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,551 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.637 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=3, num_write=0 2022-03-29T17:52:03.638 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.638 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.638 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - calling method rbd.get_flags, num_read=3, num_write=0 2022-03-29T17:52:03.638 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=4, num_write=0 2022-03-29T17:52:03.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,552 [shard 0] osd - calling method rbd.get_snapcontext, num_read=4, num_write=0 2022-03-29T17:52:03.639 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,553 [shard 0] osd - handling op omap-get-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,553 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,553 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=20 while num_read=6, num_write=0 2022-03-29T17:52:03.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,553 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,553 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.640 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,553 [shard 0] osd - calling method lock.get_info, num_read=6, num_write=0 2022-03-29T17:52:03.641 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,554 [shard 0] osd - handling op getxattr on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.641 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,554 [shard 0] osd - getxattr on obj=3:a9e8038f:::rbd_header.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.641 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,554 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=115 while num_read=7, num_write=0 2022-03-29T17:52:03.641 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,554 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.641 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,554 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:47 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_features in=24b out=16b, call rbd.get_flags in=20b out=8b, call rbd.get_snapcontext in=18b out=20b, call lock.get_info in=30b out=115b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.642 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,554 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.642 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,554 [shard 0] osd - OpSequencer::finish_op_in_order: op=29, last_started=29, last_unblocked=29, last_completed=28 2022-03-29T17:52:03.642 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,554 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4179.0:47 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_features in=24b out=16b, call rbd.get_flags in=20b out=8b, call rbd.get_snapcontext in=18b out=20b, call lock.get_info in=30b out=115b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.642 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4179.0:48 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4179.0:48 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4179.0:48 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - OpSequencer::start_op: op=30, last_started=29, last_unblocked=29, last_completed=29 2022-03-29T17:52:03.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.643 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4179.0:48 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b, call rbd.parent_overlap_get in=29b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.644 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.644 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.644 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.644 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.644 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,555 [shard 0] osd - calling method rbd.parent_get, num_read=0, num_write=0 2022-03-29T17:52:03.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.3074796 2022-03-29T17:52:03.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.645 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - handling op omap-get-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=30 while num_read=3, num_write=0 2022-03-29T17:52:03.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.646 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,556 [shard 0] osd - calling method rbd.parent_overlap_get, num_read=3, num_write=0 2022-03-29T17:52:03.647 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,557 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.647 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,557 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.3074796 2022-03-29T17:52:03.647 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,557 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.647 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,557 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.647 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,557 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=1 while num_read=5, num_write=0 2022-03-29T17:52:03.648 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,557 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.648 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,557 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:48 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b out=30b, call rbd.parent_overlap_get in=29b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.648 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.648 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - OpSequencer::finish_op_in_order: op=30, last_started=30, last_unblocked=30, last_completed=29 2022-03-29T17:52:03.649 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - client_request(id=30, detail=m=[osd_op(client.4179.0:48 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b out=30b, call rbd.parent_overlap_get in=29b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.649 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4179.0:49 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.649 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4179.0:49 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.649 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4179.0:49 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.649 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - OpSequencer::start_op: op=31, last_started=30, last_unblocked=30, last_completed=30 2022-03-29T17:52:03.650 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.650 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4179.0:49 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.650 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.650 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.650 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.651 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.651 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,558 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.651 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,559 [shard 0] osd - handling op omap-get-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.651 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,559 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.652 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,559 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.652 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,559 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:49 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.652 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,560 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.652 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,560 [shard 0] osd - OpSequencer::finish_op_in_order: op=31, last_started=31, last_unblocked=31, last_completed=30 2022-03-29T17:52:03.652 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,560 [shard 0] osd - client_request(id=31, detail=m=[osd_op(client.4179.0:49 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,560 [shard 0] osd - client_request(id=156, detail=m=[osd_op(client.4179.0:50 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,560 [shard 0] osd - client_request(id=156, detail=m=[osd_op(client.4179.0:50 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,560 [shard 0] osd - client_request(id=156, detail=m=[osd_op(client.4179.0:50 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,560 [shard 0] osd - OpSequencer::start_op: op=156, last_started=150, last_unblocked=150, last_completed=150 2022-03-29T17:52:03.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - client_request(id=156, detail=m=[osd_op(client.4179.0:50 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:03.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op call 2022-03-29T17:52:03.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - handling op call on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - handling op omap-get-vals on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:03.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,561 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:50 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:03.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,562 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,562 [shard 0] osd - OpSequencer::finish_op_in_order: op=156, last_started=156, last_unblocked=156, last_completed=150 2022-03-29T17:52:03.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,562 [shard 0] osd - client_request(id=156, detail=m=[osd_op(client.4179.0:50 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,562 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4179.0:51 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,562 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4179.0:51 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,562 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4179.0:51 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.667 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,562 [shard 0] osd - OpSequencer::start_op: op=32, last_started=31, last_unblocked=31, last_completed=31 2022-03-29T17:52:03.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4179.0:51 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.668 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - calling method rbd.image_group_get, num_read=0, num_write=0 2022-03-29T17:52:03.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,563 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=18 while num_read=1, num_write=0 2022-03-29T17:52:03.669 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:51 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b out=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - OpSequencer::finish_op_in_order: op=32, last_started=32, last_unblocked=32, last_completed=31 2022-03-29T17:52:03.670 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4179.0:51 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b out=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4179.0:52 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4179.0:52 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4179.0:52 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - OpSequencer::start_op: op=33, last_started=32, last_unblocked=32, last_completed=32 2022-03-29T17:52:03.671 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,564 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4179.0:52 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_get in=23b, call rbd.parent_overlap_get in=29b, call rbd.get_flags in=20b, call rbd.get_protection_status in=32b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.672 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - calling method rbd.snapshot_get, num_read=0, num_write=0 2022-03-29T17:52:03.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=53 while num_read=1, num_write=0 2022-03-29T17:52:03.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.673 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,565 [shard 0] osd - calling method rbd.parent_overlap_get, num_read=1, num_write=0 2022-03-29T17:52:03.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,566 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,566 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.3074796 2022-03-29T17:52:03.674 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,566 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,566 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,566 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=1 while num_read=3, num_write=0 2022-03-29T17:52:03.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,566 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,566 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.675 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,566 [shard 0] osd - calling method rbd.get_flags, num_read=3, num_write=0 2022-03-29T17:52:03.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,567 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,567 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=4, num_write=0 2022-03-29T17:52:03.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,567 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.676 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,567 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,567 [shard 0] osd - calling method rbd.get_protection_status, num_read=4, num_write=0 2022-03-29T17:52:03.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,567 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,568 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.3074796 2022-03-29T17:52:03.677 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,568 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,568 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=1 while num_read=5, num_write=0 2022-03-29T17:52:03.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,568 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,568 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:52 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_get in=23b out=53b, call rbd.parent_overlap_get in=29b out=1b, call rbd.get_flags in=20b out=8b, call rbd.get_protection_status in=32b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.678 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,568 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,568 [shard 0] osd - OpSequencer::finish_op_in_order: op=33, last_started=33, last_unblocked=33, last_completed=32 2022-03-29T17:52:03.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,568 [shard 0] osd - client_request(id=33, detail=m=[osd_op(client.4179.0:52 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_get in=23b out=53b, call rbd.parent_overlap_get in=29b out=1b, call rbd.get_flags in=20b out=8b, call rbd.get_protection_status in=32b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4179.0:53 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.679 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4179.0:53 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4179.0:53 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - OpSequencer::start_op: op=34, last_started=33, last_unblocked=33, last_completed=33 2022-03-29T17:52:03.680 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4179.0:53 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.set_protection_status in=33b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.681 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - calling method rbd.set_protection_status, num_read=0, num_write=0 2022-03-29T17:52:03.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.3074796 2022-03-29T17:52:03.682 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,569 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - handling op omap-set-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=1 2022-03-29T17:52:03.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.683 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head submitting txn 2022-03-29T17:52:03.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean calc_trim_to_aggressive limit = 18'4 2022-03-29T17:52:03.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean append_log log((0'0,18'4], crt=18'4) {18'5 (18'4) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:53 2022-03-29T17:52:02.568709+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'4 lua=0'0 crt=18'4 lcod 18'4 mlcod 18'4 active+clean add_log_entry 18'5 (18'4) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:53 2022-03-29T17:52:02.568709+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.684 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'4 lua=0'0 crt=18'5 lcod 18'4 mlcod 18'4 active+clean append_log approx pg log length = 5 2022-03-29T17:52:03.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'4 lua=0'0 crt=18'5 lcod 18'4 mlcod 18'4 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'5, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.685 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,570 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,571 [shard 0] osd - replicated_request(id=7, detail=RepRequest(from=0 req=osd_repop(client.4179.0:53 3.15 e18/14 3:a9e8038f:::rbd_header.1053953fed53:head v 18'5, mlcod=18'5) v3)) start 2022-03-29T17:52:03.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,572 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'4 (0'0,18'4] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'4 mlcod 18'4 active append_log log((0'0,18'4], crt=18'4) {18'5 (18'4) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:53 2022-03-29T17:52:02.568709+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,572 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'4 lcod 18'4 mlcod 18'4 active add_log_entry 18'5 (18'4) modify 3:a9e8038f:::rbd_header.1053953fed53:head by client.4179.0:53 2022-03-29T17:52:02.568709+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.687 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,572 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 lcod 18'4 mlcod 18'4 active append_log approx pg log length = 5 2022-03-29T17:52:03.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,572 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 lcod 18'4 mlcod 18'4 active append_log transaction_applied = 1 2022-03-29T17:52:03.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,572 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,572 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'5, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,572 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,574 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:53 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.set_protection_status in=33b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,574 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.693 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,574 [shard 0] osd - OpSequencer::finish_op_in_order: op=34, last_started=34, last_unblocked=34, last_completed=33 2022-03-29T17:52:03.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,574 [shard 0] osd - client_request(id=34, detail=m=[osd_op(client.4179.0:53 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.set_protection_status in=33b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4179.0:54 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4179.0:54 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4179.0:54 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.694 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - OpSequencer::start_op: op=35, last_started=34, last_unblocked=34, last_completed=34 2022-03-29T17:52:03.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4179.0:54 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644056448 in=22b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.695 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify 2022-03-29T17:52:03.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - handling op notify on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - do_op_notify, msg epoch: 18 2022-03-29T17:52:03.696 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.696 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,575 [shard 0] osd - start_notify adding notify(id=77309411332) 2022-03-29T17:52:03.697 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,575 [shard 0] osd - send_notify_msg for notify(id=77309411332) 2022-03-29T17:52:03.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,575 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:54 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644056448 in=22b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - OpSequencer::finish_op_in_order: op=35, last_started=35, last_unblocked=35, last_completed=34 2022-03-29T17:52:03.697 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - client_request(id=35, detail=m=[osd_op(client.4179.0:54 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify cookie 140032644056448 in=22b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4179.0:55 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4179.0:55 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4179.0:55 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.698 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - OpSequencer::start_op: op=36, last_started=35, last_unblocked=35, last_completed=35 2022-03-29T17:52:03.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4179.0:55 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.699 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op notify-ack 2022-03-29T17:52:03.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - handling op notify-ack on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - do_op_notify_ack 2022-03-29T17:52:03.700 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.700 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,576 [shard 0] osd - notify_ack watch_cookie=140032509772432, notify_id=77309411332 2022-03-29T17:52:03.700 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,576 [shard 0] osd - acking notify on watch 140032509772432,client.4179 2022-03-29T17:52:03.701 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,576 [shard 0] osd - notify_ack 2022-03-29T17:52:03.701 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,576 [shard 0] osd - send_completion -- 0 in progress watchers, timedout watchers 0 2022-03-29T17:52:03.701 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,576 [shard 0] osd - send_completion sending notify replies: notify_reply_t{watcher_gid=4179, watcher_cookie=140032509772432, bl=buffer::list(len=10, 2022-03-29T17:52:03.701 INFO:tasks.ceph.osd.0.smithi138.stderr: buffer::ptr(20~10 0x615000226bc9 in raw 0x615000226bb5 len 30 nref 4) 2022-03-29T17:52:03.701 INFO:tasks.ceph.osd.0.smithi138.stderr:)} 2022-03-29T17:52:03.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,577 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:55 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,577 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,577 [shard 0] osd - OpSequencer::finish_op_in_order: op=36, last_started=36, last_unblocked=36, last_completed=35 2022-03-29T17:52:03.702 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,577 [shard 0] osd - client_request(id=36, detail=m=[osd_op(client.4179.0:55 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {notify-ack in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,616 [shard 0] osd - client_request(id=157, detail=m=[osd_op(client.4179.0:56 3.1c 3.30a98c1c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,616 [shard 0] osd - client_request(id=157, detail=m=[osd_op(client.4179.0:56 3.1c 3.30a98c1c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,616 [shard 0] osd - client_request(id=157, detail=m=[osd_op(client.4179.0:56 3.1c 3.30a98c1c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,616 [shard 0] osd - OpSequencer::start_op: op=157, last_started=148, last_unblocked=148, last_completed=148 2022-03-29T17:52:03.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,617 [shard 0] osd - do_recover_missing check for recovery, 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,617 [shard 0] osd - client_request(id=157, detail=m=[osd_op(client.4179.0:56 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_get_id in=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,617 [shard 0] osd - with_head_obc 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,617 [shard 0] osd - with_head_obc: found 3:3831950c:::rbd_directory:head in cache 2022-03-29T17:52:03.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,617 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head - handling op call 2022-03-29T17:52:03.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,617 [shard 0] osd - handling op call on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,617 [shard 0] osd - calling method rbd.dir_get_id, num_read=0, num_write=0 2022-03-29T17:52:03.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,617 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,618 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=1, num_write=0 2022-03-29T17:52:03.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,618 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head all operations successful 2022-03-29T17:52:03.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,618 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:56 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_get_id in=23b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:3831950c:::rbd_directory:head sending reply 2022-03-29T17:52:03.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,618 [shard 0] osd - with_head_obc: released 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,618 [shard 0] osd - OpSequencer::finish_op_in_order: op=157, last_started=157, last_unblocked=157, last_completed=148 2022-03-29T17:52:03.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,618 [shard 0] osd - client_request(id=157, detail=m=[osd_op(client.4179.0:56 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_get_id in=23b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,621 [shard 0] osd - client_request(id=158, detail=m=[osd_op(client.4179.0:57 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,621 [shard 0] osd - client_request(id=158, detail=m=[osd_op(client.4179.0:57 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,621 [shard 0] osd - client_request(id=158, detail=m=[osd_op(client.4179.0:57 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,621 [shard 0] osd - OpSequencer::start_op: op=158, last_started=156, last_unblocked=156, last_completed=156 2022-03-29T17:52:03.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,621 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,621 [shard 0] osd - client_request(id=158, detail=m=[osd_op(client.4179.0:57 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,621 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,621 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:03.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,622 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op call 2022-03-29T17:52:03.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,622 [shard 0] osd - handling op call on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,622 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,622 [shard 0] osd - handling op omap-get-vals on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,622 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,622 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:03.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,622 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:57 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:03.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,622 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,623 [shard 0] osd - OpSequencer::finish_op_in_order: op=158, last_started=158, last_unblocked=158, last_completed=156 2022-03-29T17:52:03.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,623 [shard 0] osd - client_request(id=158, detail=m=[osd_op(client.4179.0:57 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - client_request(id=159, detail=m=[osd_op(client.4179.0:58 3.1c 3.30a98c1c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - client_request(id=159, detail=m=[osd_op(client.4179.0:58 3.1c 3.30a98c1c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - client_request(id=159, detail=m=[osd_op(client.4179.0:58 3.1c 3.30a98c1c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - OpSequencer::start_op: op=159, last_started=157, last_unblocked=157, last_completed=157 2022-03-29T17:52:03.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - do_recover_missing check for recovery, 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - client_request(id=159, detail=m=[osd_op(client.4179.0:58 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_get_name in=31b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - with_head_obc 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - with_head_obc: found 3:3831950c:::rbd_directory:head in cache 2022-03-29T17:52:03.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,625 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head - handling op call 2022-03-29T17:52:03.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,626 [shard 0] osd - handling op call on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,626 [shard 0] osd - calling method rbd.dir_get_name, num_read=0, num_write=0 2022-03-29T17:52:03.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,626 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,626 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=10 while num_read=1, num_write=0 2022-03-29T17:52:03.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,626 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head all operations successful 2022-03-29T17:52:03.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,626 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:58 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_get_name in=31b out=10b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:3831950c:::rbd_directory:head sending reply 2022-03-29T17:52:03.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,627 [shard 0] osd - with_head_obc: released 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,627 [shard 0] osd - OpSequencer::finish_op_in_order: op=159, last_started=159, last_unblocked=159, last_completed=157 2022-03-29T17:52:03.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,627 [shard 0] osd - client_request(id=159, detail=m=[osd_op(client.4179.0:58 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_get_name in=31b out=10b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.735 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,627 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4179.0:59 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,627 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4179.0:59 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,627 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4179.0:59 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,627 [shard 0] osd - OpSequencer::start_op: op=37, last_started=36, last_unblocked=36, last_completed=36 2022-03-29T17:52:03.736 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4179.0:59 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b, call rbd.get_object_prefix in=20b, call rbd.get_features in=24b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.737 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - calling method rbd.get_size, num_read=0, num_write=0 2022-03-29T17:52:03.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,628 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.738 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,629 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=9 while num_read=2, num_write=0 2022-03-29T17:52:03.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,629 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,629 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,629 [shard 0] osd - calling method rbd.get_object_prefix, num_read=2, num_write=0 2022-03-29T17:52:03.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,629 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.739 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,630 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=25 while num_read=3, num_write=0 2022-03-29T17:52:03.740 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,630 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.740 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,630 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.740 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,630 [shard 0] osd - calling method rbd.get_features, num_read=3, num_write=0 2022-03-29T17:52:03.740 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,630 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.741 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,630 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=4, num_write=0 2022-03-29T17:52:03.741 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,630 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.741 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,630 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:59 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_object_prefix in=20b out=25b, call rbd.get_features in=24b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.741 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - OpSequencer::finish_op_in_order: op=37, last_started=37, last_unblocked=37, last_completed=36 2022-03-29T17:52:03.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - client_request(id=37, detail=m=[osd_op(client.4179.0:59 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_object_prefix in=20b out=25b, call rbd.get_features in=24b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4179.0:60 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4179.0:60 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.742 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4179.0:60 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.743 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - OpSequencer::start_op: op=38, last_started=37, last_unblocked=37, last_completed=37 2022-03-29T17:52:03.743 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.743 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4179.0:60 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_create_timestamp in=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.743 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.744 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.744 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.744 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.744 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,631 [shard 0] osd - calling method rbd.get_create_timestamp, num_read=0, num_write=0 2022-03-29T17:52:03.744 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,632 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,632 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:03.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,632 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,632 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:60 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_create_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,632 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.745 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,632 [shard 0] osd - OpSequencer::finish_op_in_order: op=38, last_started=38, last_unblocked=38, last_completed=37 2022-03-29T17:52:03.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,632 [shard 0] osd - client_request(id=38, detail=m=[osd_op(client.4179.0:60 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_create_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4179.0:61 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4179.0:61 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.746 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4179.0:61 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - OpSequencer::start_op: op=39, last_started=38, last_unblocked=38, last_completed=38 2022-03-29T17:52:03.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4179.0:61 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_access_timestamp in=23b, call rbd.get_modify_timestamp in=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.747 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,633 [shard 0] osd - calling method rbd.get_access_timestamp, num_read=0, num_write=0 2022-03-29T17:52:03.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,634 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.748 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,634 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:03.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,634 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,634 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,634 [shard 0] osd - calling method rbd.get_modify_timestamp, num_read=1, num_write=0 2022-03-29T17:52:03.749 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,634 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.750 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,634 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=2, num_write=0 2022-03-29T17:52:03.750 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,634 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.750 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:61 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_access_timestamp in=23b out=8b, call rbd.get_modify_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.750 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.750 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - OpSequencer::finish_op_in_order: op=39, last_started=39, last_unblocked=39, last_completed=38 2022-03-29T17:52:03.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - client_request(id=39, detail=m=[osd_op(client.4179.0:61 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_access_timestamp in=23b out=8b, call rbd.get_modify_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4179.0:62 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4179.0:62 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.751 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4179.0:62 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - OpSequencer::start_op: op=40, last_started=39, last_unblocked=39, last_completed=39 2022-03-29T17:52:03.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4179.0:62 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_data_pool in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.752 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,635 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,636 [shard 0] osd - calling method rbd.get_data_pool, num_read=0, num_write=0 2022-03-29T17:52:03.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,636 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.753 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,636 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:03.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,636 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,636 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:62 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_data_pool in=16b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,636 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.754 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,636 [shard 0] osd - OpSequencer::finish_op_in_order: op=40, last_started=40, last_unblocked=40, last_completed=39 2022-03-29T17:52:03.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,637 [shard 0] osd - client_request(id=40, detail=m=[osd_op(client.4179.0:62 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_data_pool in=16b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,637 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4179.0:63 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,637 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4179.0:63 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,637 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4179.0:63 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.755 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,637 [shard 0] osd - OpSequencer::start_op: op=41, last_started=40, last_unblocked=40, last_completed=40 2022-03-29T17:52:03.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,637 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,637 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4179.0:63 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b, call rbd.get_features in=24b, call rbd.get_flags in=20b, call rbd.get_snapcontext in=18b, call lock.get_info in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,637 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.756 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.757 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.757 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.757 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - calling method rbd.get_size, num_read=0, num_write=0 2022-03-29T17:52:03.757 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.757 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=9 while num_read=2, num_write=0 2022-03-29T17:52:03.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.758 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,638 [shard 0] osd - calling method rbd.get_features, num_read=2, num_write=0 2022-03-29T17:52:03.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,639 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,639 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=3, num_write=0 2022-03-29T17:52:03.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,639 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.759 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,639 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,639 [shard 0] osd - calling method rbd.get_flags, num_read=3, num_write=0 2022-03-29T17:52:03.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,640 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,640 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=4, num_write=0 2022-03-29T17:52:03.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,640 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.760 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,640 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,640 [shard 0] osd - calling method rbd.get_snapcontext, num_read=4, num_write=0 2022-03-29T17:52:03.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,640 [shard 0] osd - handling op omap-get-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,640 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=20 while num_read=6, num_write=0 2022-03-29T17:52:03.761 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - calling method lock.get_info, num_read=6, num_write=0 2022-03-29T17:52:03.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - handling op getxattr on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.762 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - getxattr on obj=3:a9e8038f:::rbd_header.1053953fed53:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=115 while num_read=7, num_write=0 2022-03-29T17:52:03.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,641 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:63 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_features in=24b out=16b, call rbd.get_flags in=20b out=8b, call rbd.get_snapcontext in=18b out=20b, call lock.get_info in=30b out=115b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.763 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - OpSequencer::finish_op_in_order: op=41, last_started=41, last_unblocked=41, last_completed=40 2022-03-29T17:52:03.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - client_request(id=41, detail=m=[osd_op(client.4179.0:63 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.get_size in=19b out=9b, call rbd.get_features in=24b out=16b, call rbd.get_flags in=20b out=8b, call rbd.get_snapcontext in=18b out=20b, call lock.get_info in=30b out=115b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4179.0:64 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4179.0:64 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4179.0:64 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.764 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - OpSequencer::start_op: op=42, last_started=41, last_unblocked=41, last_completed=41 2022-03-29T17:52:03.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4179.0:64 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b, call rbd.parent_overlap_get in=29b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.765 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,642 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,643 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,643 [shard 0] osd - calling method rbd.parent_get, num_read=0, num_write=0 2022-03-29T17:52:03.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,643 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.766 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,643 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.5687099 2022-03-29T17:52:03.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,643 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,643 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,643 [shard 0] osd - handling op omap-get-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,644 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=30 while num_read=3, num_write=0 2022-03-29T17:52:03.767 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,644 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,644 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,644 [shard 0] osd - calling method rbd.parent_overlap_get, num_read=3, num_write=0 2022-03-29T17:52:03.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,644 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,644 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.5687099 2022-03-29T17:52:03.768 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,644 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,644 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=1 while num_read=5, num_write=0 2022-03-29T17:52:03.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:64 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b out=30b, call rbd.parent_overlap_get in=29b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.769 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - OpSequencer::finish_op_in_order: op=42, last_started=42, last_unblocked=42, last_completed=41 2022-03-29T17:52:03.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - client_request(id=42, detail=m=[osd_op(client.4179.0:64 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.parent_get in=13b out=30b, call rbd.parent_overlap_get in=29b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4179.0:65 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.770 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4179.0:65 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4179.0:65 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - OpSequencer::start_op: op=43, last_started=42, last_unblocked=42, last_completed=42 2022-03-29T17:52:03.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4179.0:65 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.771 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,645 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,646 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,646 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,646 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,646 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.772 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,646 [shard 0] osd - handling op omap-get-vals on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,646 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,646 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,646 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:65 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.773 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - OpSequencer::finish_op_in_order: op=43, last_started=43, last_unblocked=43, last_completed=42 2022-03-29T17:52:03.774 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - client_request(id=43, detail=m=[osd_op(client.4179.0:65 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - client_request(id=160, detail=m=[osd_op(client.4179.0:66 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - client_request(id=160, detail=m=[osd_op(client.4179.0:66 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - client_request(id=160, detail=m=[osd_op(client.4179.0:66 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - OpSequencer::start_op: op=160, last_started=158, last_unblocked=158, last_completed=158 2022-03-29T17:52:03.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - client_request(id=160, detail=m=[osd_op(client.4179.0:66 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,647 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,648 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:03.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,648 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op call 2022-03-29T17:52:03.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,648 [shard 0] osd - handling op call on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,648 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,648 [shard 0] osd - handling op omap-get-vals on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,648 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,648 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:03.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,648 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:66 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:03.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - OpSequencer::finish_op_in_order: op=160, last_started=160, last_unblocked=160, last_completed=158 2022-03-29T17:52:03.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - client_request(id=160, detail=m=[osd_op(client.4179.0:66 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.785 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4179.0:67 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.785 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4179.0:67 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4179.0:67 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - OpSequencer::start_op: op=44, last_started=43, last_unblocked=43, last_completed=43 2022-03-29T17:52:03.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4179.0:67 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.786 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,649 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.787 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - calling method rbd.image_group_get, num_read=0, num_write=0 2022-03-29T17:52:03.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=18 while num_read=1, num_write=0 2022-03-29T17:52:03.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:67 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b out=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.788 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,650 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - OpSequencer::finish_op_in_order: op=44, last_started=44, last_unblocked=44, last_completed=43 2022-03-29T17:52:03.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - client_request(id=44, detail=m=[osd_op(client.4179.0:67 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.image_group_get in=18b out=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4179.0:68 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4179.0:68 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.789 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4179.0:68 3.15 3.f1c01795 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - OpSequencer::start_op: op=45, last_started=44, last_unblocked=44, last_completed=44 2022-03-29T17:52:03.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4179.0:68 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_get in=23b, call rbd.parent_overlap_get in=29b, call rbd.get_flags in=20b, call rbd.get_protection_status in=32b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.790 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - with_head_obc 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - with_head_obc: found 3:a9e8038f:::rbd_header.1053953fed53:head in cache 2022-03-29T17:52:03.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,651 [shard 0] osd - calling method rbd.snapshot_get, num_read=0, num_write=0 2022-03-29T17:52:03.791 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,652 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,652 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=53 while num_read=1, num_write=0 2022-03-29T17:52:03.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,652 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,652 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,652 [shard 0] osd - calling method rbd.parent_overlap_get, num_read=1, num_write=0 2022-03-29T17:52:03.792 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,652 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,652 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.5687099 2022-03-29T17:52:03.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,652 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,653 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,653 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=1 while num_read=3, num_write=0 2022-03-29T17:52:03.793 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,653 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,653 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,653 [shard 0] osd - calling method rbd.get_flags, num_read=3, num_write=0 2022-03-29T17:52:03.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,653 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.794 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,653 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=4, num_write=0 2022-03-29T17:52:03.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,653 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head - handling op call 2022-03-29T17:52:03.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - handling op call on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - calling method rbd.get_protection_status, num_read=4, num_write=0 2022-03-29T17:52:03.795 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - handling op stat on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.5687099 2022-03-29T17:52:03.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=1 while num_read=5, num_write=0 2022-03-29T17:52:03.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - do_osd_ops_execute: object 3:a9e8038f:::rbd_header.1053953fed53:head all operations successful 2022-03-29T17:52:03.796 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:68 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_get in=23b out=53b, call rbd.parent_overlap_get in=29b out=1b, call rbd.get_flags in=20b out=8b, call rbd.get_protection_status in=32b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:a9e8038f:::rbd_header.1053953fed53:head sending reply 2022-03-29T17:52:03.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,654 [shard 0] osd - with_head_obc: released 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:03.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - OpSequencer::finish_op_in_order: op=45, last_started=45, last_unblocked=45, last_completed=44 2022-03-29T17:52:03.797 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - client_request(id=45, detail=m=[osd_op(client.4179.0:68 3.15 3:a9e8038f:::rbd_header.1053953fed53:head {call rbd.snapshot_get in=23b out=53b, call rbd.parent_overlap_get in=29b out=1b, call rbd.get_flags in=20b out=8b, call rbd.get_protection_status in=32b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:69 3.7 3.5941b7e7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.801 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:69 3.7 3.5941b7e7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:69 3.7 3.5941b7e7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - OpSequencer::start_op: op=4, last_started=3, last_unblocked=3, last_completed=3 2022-03-29T17:52:03.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - do_recover_missing check for recovery, 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.802 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:69 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {call rbd.object_map_load in=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - with_head_obc 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,655 [shard 0] osd - with_head_obc: found 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head in cache 2022-03-29T17:52:03.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,656 [shard 0] osd - do_osd_ops_execute: object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head - handling op call 2022-03-29T17:52:03.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,656 [shard 0] osd - handling op call on object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.803 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,656 [shard 0] osd - calling method rbd.object_map_load, num_read=0, num_write=0 2022-03-29T17:52:03.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,656 [shard 0] osd - handling op stat on object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,656 [shard 0] osd - stat os.oi.size=35, os.oi.mtime=1648576322.388366 2022-03-29T17:52:03.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,656 [shard 0] osd - handling op sync_read on object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.804 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - read: data length: 35 2022-03-29T17:52:03.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=23 while num_read=1, num_write=0 2022-03-29T17:52:03.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - do_osd_ops_execute: object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head all operations successful 2022-03-29T17:52:03.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:69 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {call rbd.object_map_load in=18b out=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head sending reply 2022-03-29T17:52:03.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - with_head_obc: released 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.805 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - OpSequencer::finish_op_in_order: op=4, last_started=4, last_unblocked=4, last_completed=3 2022-03-29T17:52:03.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - client_request(id=4, detail=m=[osd_op(client.4179.0:69 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {call rbd.object_map_load in=18b out=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:70 3.7 3.5941b7e7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:70 3.7 3.5941b7e7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.806 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,657 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:70 3.7 3.5941b7e7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - OpSequencer::start_op: op=5, last_started=4, last_unblocked=4, last_completed=4 2022-03-29T17:52:03.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - do_recover_missing check for recovery, 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:70 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {call rbd.object_map_load in=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - with_head_obc 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.807 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - with_head_obc: found 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head in cache 2022-03-29T17:52:03.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - do_osd_ops_execute: object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head - handling op call 2022-03-29T17:52:03.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - handling op call on object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - calling method rbd.object_map_load, num_read=0, num_write=0 2022-03-29T17:52:03.808 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - handling op stat on object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - stat os.oi.size=35, os.oi.mtime=1648576322.388366 2022-03-29T17:52:03.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - handling op sync_read on object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,658 [shard 0] osd - read: data length: 35 2022-03-29T17:52:03.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=23 while num_read=1, num_write=0 2022-03-29T17:52:03.809 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - do_osd_ops_execute: object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head all operations successful 2022-03-29T17:52:03.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:70 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {call rbd.object_map_load in=18b out=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head sending reply 2022-03-29T17:52:03.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - with_head_obc: released 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head 2022-03-29T17:52:03.810 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - OpSequencer::finish_op_in_order: op=5, last_started=5, last_unblocked=5, last_completed=4 2022-03-29T17:52:03.813 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - client_request(id=161, detail=m=[osd_op(client.4179.0:71 3.1c 3.1b932e9c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - client_request(id=161, detail=m=[osd_op(client.4179.0:71 3.1c 3.1b932e9c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - client_request(id=161, detail=m=[osd_op(client.4179.0:71 3.1c 3.1b932e9c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.814 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - OpSequencer::start_op: op=161, last_started=159, last_unblocked=159, last_completed=159 2022-03-29T17:52:03.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - do_recover_missing check for recovery, 3:3974c9d8:::image2.rbd:head 2022-03-29T17:52:03.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - client_request(id=161, detail=m=[osd_op(client.4179.0:71 3.1c 3:3974c9d8:::image2.rbd:head {stat} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - with_head_obc 3:3974c9d8:::image2.rbd:head 2022-03-29T17:52:03.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - with_head_obc: cache miss on 3:3974c9d8:::image2.rbd:head 2022-03-29T17:52:03.815 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - load_metadata: object 3:3974c9d8:::image2.rbd:head doesn't exist, returning empty metadata 2022-03-29T17:52:03.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - load_head_obc: loaded obs 3:3974c9d8:::image2.rbd:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:3974c9d8:::image2.rbd:head 2022-03-29T17:52:03.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - load_head_obc: returning obc 3:3974c9d8:::image2.rbd:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:3974c9d8:::image2.rbd:head 2022-03-29T17:52:03.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - do_osd_ops_execute: object 3:3974c9d8:::image2.rbd:head - handling op stat 2022-03-29T17:52:03.816 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - handling op stat on object 3:3974c9d8:::image2.rbd:head 2022-03-29T17:52:03.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - stat object does not exist 2022-03-29T17:52:03.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - rollback_obc_if_modified: object 3:3974c9d8:::image2.rbd:head got error generic:2, need_rollback=false 2022-03-29T17:52:03.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - with_head_obc: released 3:3974c9d8:::image2.rbd:head 2022-03-29T17:52:03.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - OpSequencer::finish_op_in_order: op=161, last_started=161, last_unblocked=161, last_completed=159 2022-03-29T17:52:03.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,660 [shard 0] osd - client_request(id=161, detail=m=[osd_op(client.4179.0:71 3.1c 3:3974c9d8:::image2.rbd:head {stat} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - client_request(id=162, detail=m=[osd_op(client.4179.0:72 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - client_request(id=162, detail=m=[osd_op(client.4179.0:72 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - client_request(id=162, detail=m=[osd_op(client.4179.0:72 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - OpSequencer::start_op: op=162, last_started=160, last_unblocked=160, last_completed=160 2022-03-29T17:52:03.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - client_request(id=162, detail=m=[osd_op(client.4179.0:72 3.1a 3:5cea7035:::rbd_info:head {read 0~0} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:03.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op read 2022-03-29T17:52:03.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,661 [shard 0] osd - handling op read on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.820 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,659 [shard 0] osd - client_request(id=5, detail=m=[osd_op(client.4179.0:70 3.7 3:e7ed829a:::rbd_object_map.1053953fed53.0000000000000004:head {call rbd.object_map_load in=18b out=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - read: data length: 19 2022-03-29T17:52:03.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:03.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:72 3.1a 3:5cea7035:::rbd_info:head {read 0~0 out=19b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:03.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - OpSequencer::finish_op_in_order: op=162, last_started=162, last_unblocked=162, last_completed=160 2022-03-29T17:52:03.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - client_request(id=162, detail=m=[osd_op(client.4179.0:72 3.1a 3:5cea7035:::rbd_info:head {read 0~0 out=19b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - client_request(id=163, detail=m=[osd_op(client.4179.0:73 3.1c 3.30a98c1c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - client_request(id=163, detail=m=[osd_op(client.4179.0:73 3.1c 3.30a98c1c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - client_request(id=163, detail=m=[osd_op(client.4179.0:73 3.1c 3.30a98c1c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - OpSequencer::start_op: op=163, last_started=161, last_unblocked=161, last_completed=161 2022-03-29T17:52:03.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - do_recover_missing check for recovery, 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,662 [shard 0] osd - client_request(id=163, detail=m=[osd_op(client.4179.0:73 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_add_image in=41b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,663 [shard 0] osd - with_head_obc 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,663 [shard 0] osd - with_head_obc: found 3:3831950c:::rbd_directory:head in cache 2022-03-29T17:52:03.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,663 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head - handling op call 2022-03-29T17:52:03.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,663 [shard 0] osd - handling op call on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,663 [shard 0] osd - calling method rbd.dir_add_image, num_read=0, num_write=0 2022-03-29T17:52:03.827 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,663 [shard 0] osd - handling op create on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,663 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,663 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - handling op omap-set-vals on object 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=2, num_write=2 2022-03-29T17:52:03.828 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head all operations successful 2022-03-29T17:52:03.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - do_osd_ops_execute: object 3:3831950c:::rbd_directory:head submitting txn 2022-03-29T17:52:03.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean calc_trim_to_aggressive limit = 17'1 2022-03-29T17:52:03.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=17'1 mlcod 17'1 active+clean append_log log((0'0,17'1], crt=17'1) {18'2 (17'1) modify 3:3831950c:::rbd_directory:head by client.4179.0:73 2022-03-29T17:52:02.662429+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'1 lua=0'0 crt=17'1 lcod 17'1 mlcod 17'1 active+clean add_log_entry 18'2 (17'1) modify 3:3831950c:::rbd_directory:head by client.4179.0:73 2022-03-29T17:52:02.662429+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'1 lua=0'0 crt=18'2 lcod 17'1 mlcod 17'1 active+clean append_log approx pg log length = 2 2022-03-29T17:52:03.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=17'1 lua=0'0 crt=18'2 lcod 17'1 mlcod 17'1 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,664 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.831 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,665 [shard 0] osd - replicated_request(id=103, detail=RepRequest(from=1 req=osd_repop(client.4179.0:73 3.1c e18/14 3:3831950c:::rbd_directory:head v 18'2, mlcod=18'2) v3)) start 2022-03-29T17:52:03.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,666 [shard 0] osd - pg_epoch 18 pg[3.1c( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active append_log log((0'0,17'1], crt=17'1) {18'2 (17'1) modify 3:3831950c:::rbd_directory:head by client.4179.0:73 2022-03-29T17:52:02.662429+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,666 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 lcod 17'1 mlcod 17'1 active add_log_entry 18'2 (17'1) modify 3:3831950c:::rbd_directory:head by client.4179.0:73 2022-03-29T17:52:02.662429+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,666 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'2 lcod 17'1 mlcod 17'1 active append_log approx pg log length = 2 2022-03-29T17:52:03.832 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,666 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'2 lcod 17'1 mlcod 17'1 active append_log transaction_applied = 1 2022-03-29T17:52:03.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,666 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,666 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.833 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,666 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,668 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:73 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_add_image in=41b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:3831950c:::rbd_directory:head sending reply 2022-03-29T17:52:03.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,668 [shard 0] osd - with_head_obc: released 3:3831950c:::rbd_directory:head 2022-03-29T17:52:03.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,668 [shard 0] osd - OpSequencer::finish_op_in_order: op=163, last_started=163, last_unblocked=163, last_completed=161 2022-03-29T17:52:03.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,668 [shard 0] osd - client_request(id=163, detail=m=[osd_op(client.4179.0:73 3.1c 3:3831950c:::rbd_directory:head {call rbd.dir_add_image in=41b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,668 [shard 0] osd - client_request(id=164, detail=m=[osd_op(client.4179.0:74 3.1c 3.a4cfcbc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,668 [shard 0] osd - client_request(id=164, detail=m=[osd_op(client.4179.0:74 3.1c 3.a4cfcbc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,668 [shard 0] osd - client_request(id=164, detail=m=[osd_op(client.4179.0:74 3.1c 3.a4cfcbc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,668 [shard 0] osd - OpSequencer::start_op: op=164, last_started=163, last_unblocked=163, last_completed=163 2022-03-29T17:52:03.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - do_recover_missing check for recovery, 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - client_request(id=164, detail=m=[osd_op(client.4179.0:74 3.1c 3:3d3f3250:::rbd_id.image2:head {create, call rbd.set_id in=24b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - with_head_obc 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - with_head_obc: cache miss on 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - load_metadata: object 3:3d3f3250:::rbd_id.image2:head doesn't exist, returning empty metadata 2022-03-29T17:52:03.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - load_head_obc: loaded obs 3:3d3f3250:::rbd_id.image2:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - load_head_obc: returning obc 3:3d3f3250:::rbd_id.image2:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - do_osd_ops_execute: object 3:3d3f3250:::rbd_id.image2:head - handling op create 2022-03-29T17:52:03.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - handling op create on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - do_osd_ops_execute: object 3:3d3f3250:::rbd_id.image2:head - handling op call 2022-03-29T17:52:03.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - handling op call on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,669 [shard 0] osd - calling method rbd.set_id, num_read=0, num_write=1 2022-03-29T17:52:03.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - handling op stat on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=0.0 2022-03-29T17:52:03.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - handling op stat on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=0.0 2022-03-29T17:52:03.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - handling op write on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.842 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=0, num_write=2 2022-03-29T17:52:03.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - do_osd_ops_execute: object 3:3d3f3250:::rbd_id.image2:head all operations successful 2022-03-29T17:52:03.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - do_osd_ops_execute: object 3:3d3f3250:::rbd_id.image2:head submitting txn 2022-03-29T17:52:03.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'2 mlcod 18'2 active+clean calc_trim_to_aggressive limit = 18'2 2022-03-29T17:52:03.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'2 mlcod 18'2 active+clean append_log log((0'0,18'2], crt=18'2) {18'3 (0'0) modify 3:3d3f3250:::rbd_id.image2:head by client.4179.0:74 2022-03-29T17:52:02.668514+0000 0 ObjectCleanRegions clean_offsets: [15~18446744073709551600], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.843 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=18'2 lua=0'0 crt=18'2 lcod 18'2 mlcod 18'2 active+clean add_log_entry 18'3 (0'0) modify 3:3d3f3250:::rbd_id.image2:head by client.4179.0:74 2022-03-29T17:52:02.668514+0000 0 ObjectCleanRegions clean_offsets: [15~18446744073709551600], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=18'2 lua=0'0 crt=18'3 lcod 18'2 mlcod 18'2 active+clean append_log approx pg log length = 3 2022-03-29T17:52:03.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 luod=18'2 lua=0'0 crt=18'3 lcod 18'2 mlcod 18'2 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.844 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,670 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,708 [shard 0] osd - replicated_request(id=104, detail=RepRequest(from=1 req=osd_repop(client.4179.0:74 3.1c e18/14 3:3d3f3250:::rbd_id.image2:head v 18'3, mlcod=18'3) v3)) start 2022-03-29T17:52:03.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,709 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'2 (0'0,18'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'2 mlcod 18'2 active append_log log((0'0,18'2], crt=18'2) {18'3 (0'0) modify 3:3d3f3250:::rbd_id.image2:head by client.4179.0:74 2022-03-29T17:52:02.668514+0000 0 ObjectCleanRegions clean_offsets: [15~18446744073709551600], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,709 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'2 lcod 18'2 mlcod 18'2 active add_log_entry 18'3 (0'0) modify 3:3d3f3250:::rbd_id.image2:head by client.4179.0:74 2022-03-29T17:52:02.668514+0000 0 ObjectCleanRegions clean_offsets: [15~18446744073709551600], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.846 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,709 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 lcod 18'2 mlcod 18'2 active append_log approx pg log length = 3 2022-03-29T17:52:03.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,709 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 lcod 18'2 mlcod 18'2 active append_log transaction_applied = 1 2022-03-29T17:52:03.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,709 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,709 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.847 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,709 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,749 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:74 3.1c 3:3d3f3250:::rbd_id.image2:head {create, call rbd.set_id in=24b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:3d3f3250:::rbd_id.image2:head sending reply 2022-03-29T17:52:03.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,749 [shard 0] osd - with_head_obc: released 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,750 [shard 0] osd - OpSequencer::finish_op_in_order: op=164, last_started=164, last_unblocked=164, last_completed=163 2022-03-29T17:52:03.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,750 [shard 0] osd - client_request(id=164, detail=m=[osd_op(client.4179.0:74 3.1c 3:3d3f3250:::rbd_id.image2:head {create, call rbd.set_id in=24b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,750 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4179.0:75 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,750 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4179.0:75 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,750 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4179.0:75 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.852 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,750 [shard 0] osd - OpSequencer::start_op: op=46, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:03.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,750 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4179.0:75 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {create, call rbd.create in=58b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.853 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - with_head_obc: cache miss on 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - load_metadata: object 3:5741e4f5:::rbd_header.1053eb73922:head doesn't exist, returning empty metadata 2022-03-29T17:52:03.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - load_head_obc: loaded obs 3:5741e4f5:::rbd_header.1053eb73922:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - load_head_obc: returning obc 3:5741e4f5:::rbd_header.1053eb73922:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.854 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op create 2022-03-29T17:52:03.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - handling op create on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,751 [shard 0] osd - calling method rbd.create, num_read=0, num_write=1 2022-03-29T17:52:03.855 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - handling op omap-set-vals on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=1, num_write=2 2022-03-29T17:52:03.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.856 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head submitting txn 2022-03-29T17:52:03.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:03.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {18'1 (0'0) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:75 2022-03-29T17:52:02.750055+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 18'1 (0'0) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:75 2022-03-29T17:52:02.750055+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:03.857 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.858 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,752 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,753 [shard 0] osd - replicated_request(id=8, detail=RepRequest(from=0 req=osd_repop(client.4179.0:75 3.a e18/14 3:5741e4f5:::rbd_header.1053eb73922:head v 18'1, mlcod=18'1) v3)) start 2022-03-29T17:52:03.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,754 [shard 0] osd - pg_epoch 18 pg[3.a( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {18'1 (0'0) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:75 2022-03-29T17:52:02.750055+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,754 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 18'1 (0'0) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:75 2022-03-29T17:52:02.750055+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,754 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:03.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,754 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:03.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,754 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,754 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,754 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.861 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,794 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:75 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {create, call rbd.create in=58b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,795 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:76 3.e 3.a57f008e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,795 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:76 3.e 3.a57f008e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,795 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:76 3.e 3.a57f008e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,795 [shard 0] osd - OpSequencer::start_op: op=6, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:03.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,795 [shard 0] osd - do_recover_missing check for recovery, 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,795 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:76 3.e 3:7100fea5:::rbd_object_map.1053eb73922:head {call rbd.object_map_resize in=29b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,795 [shard 0] osd - with_head_obc 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,795 [shard 0] osd - with_head_obc: cache miss on 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - load_metadata: object 3:7100fea5:::rbd_object_map.1053eb73922:head doesn't exist, returning empty metadata 2022-03-29T17:52:03.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - load_head_obc: loaded obs 3:7100fea5:::rbd_object_map.1053eb73922:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - load_head_obc: returning obc 3:7100fea5:::rbd_object_map.1053eb73922:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - do_osd_ops_execute: object 3:7100fea5:::rbd_object_map.1053eb73922:head - handling op call 2022-03-29T17:52:03.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - handling op call on object 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.867 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - calling method rbd.object_map_resize, num_read=0, num_write=0 2022-03-29T17:52:03.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - handling op stat on object 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - stat object does not exist 2022-03-29T17:52:03.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,796 [shard 0] osd - handling op writefull on object 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=0, num_write=1 2022-03-29T17:52:03.868 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - do_osd_ops_execute: object 3:7100fea5:::rbd_object_map.1053eb73922:head all operations successful 2022-03-29T17:52:03.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - do_osd_ops_execute: object 3:7100fea5:::rbd_object_map.1053eb73922:head submitting txn 2022-03-29T17:52:03.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean calc_trim_to_aggressive limit = 0'0 2022-03-29T17:52:03.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean append_log log((0'0,0'0], crt=0'0) {18'1 (0'0) modify 3:7100fea5:::rbd_object_map.1053eb73922:head by client.4179.0:76 2022-03-29T17:52:02.794943+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean add_log_entry 18'1 (0'0) modify 3:7100fea5:::rbd_object_map.1053eb73922:head by client.4179.0:76 2022-03-29T17:52:02.794943+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active+clean append_log approx pg log length = 1 2022-03-29T17:52:03.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,797 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,798 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,798 [shard 0] osd - OpSequencer::finish_op_in_order: op=46, last_started=46, last_unblocked=46, last_completed=0 2022-03-29T17:52:03.872 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,798 [shard 0] osd - client_request(id=46, detail=m=[osd_op(client.4179.0:75 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {create, call rbd.create in=58b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,799 [shard 0] osd - replicated_request(id=105, detail=RepRequest(from=2 req=osd_repop(client.4179.0:76 3.e e18/14 3:7100fea5:::rbd_object_map.1053eb73922:head v 18'1, mlcod=18'1) v3)) start 2022-03-29T17:52:03.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,799 [shard 0] osd - pg_epoch 18 pg[3.e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 crt=0'0 mlcod 0'0 active append_log log((0'0,0'0], crt=0'0) {18'1 (0'0) modify 3:7100fea5:::rbd_object_map.1053eb73922:head by client.4179.0:76 2022-03-29T17:52:02.794943+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,799 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active add_log_entry 18'1 (0'0) modify 3:7100fea5:::rbd_object_map.1053eb73922:head by client.4179.0:76 2022-03-29T17:52:02.794943+0000 0 ObjectCleanRegions clean_offsets: [35~18446744073709551580], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,799 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active append_log approx pg log length = 1 2022-03-29T17:52:03.873 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,799 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 0'0 mlcod 0'0 active append_log transaction_applied = 1 2022-03-29T17:52:03.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,799 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,799 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'1, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.874 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,799 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,801 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:76 3.e 3:7100fea5:::rbd_object_map.1053eb73922:head {call rbd.object_map_resize in=29b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:7100fea5:::rbd_object_map.1053eb73922:head sending reply 2022-03-29T17:52:03.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,802 [shard 0] osd - with_head_obc: released 3:7100fea5:::rbd_object_map.1053eb73922:head 2022-03-29T17:52:03.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,802 [shard 0] osd - OpSequencer::finish_op_in_order: op=6, last_started=6, last_unblocked=6, last_completed=0 2022-03-29T17:52:03.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:02,802 [shard 0] osd - client_request(id=6, detail=m=[osd_op(client.4179.0:76 3.e 3:7100fea5:::rbd_object_map.1053eb73922:head {call rbd.object_map_resize in=29b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,804 [shard 0] osd - client_request(id=165, detail=m=[osd_op(client.4179.0:77 3.1c 3.a4cfcbc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,804 [shard 0] osd - client_request(id=165, detail=m=[osd_op(client.4179.0:77 3.1c 3.a4cfcbc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,804 [shard 0] osd - client_request(id=165, detail=m=[osd_op(client.4179.0:77 3.1c 3.a4cfcbc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,804 [shard 0] osd - OpSequencer::start_op: op=165, last_started=164, last_unblocked=164, last_completed=164 2022-03-29T17:52:03.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - do_recover_missing check for recovery, 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - client_request(id=165, detail=m=[osd_op(client.4179.0:77 3.1c 3:3d3f3250:::rbd_id.image2:head {stat} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - with_head_obc 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - with_head_obc: found 3:3d3f3250:::rbd_id.image2:head in cache 2022-03-29T17:52:03.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - do_osd_ops_execute: object 3:3d3f3250:::rbd_id.image2:head - handling op stat 2022-03-29T17:52:03.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - handling op stat on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - stat os.oi.size=15, os.oi.mtime=1648576322.6685143 2022-03-29T17:52:03.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - do_osd_ops_execute: object 3:3d3f3250:::rbd_id.image2:head all operations successful 2022-03-29T17:52:03.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:77 3.1c 3:3d3f3250:::rbd_id.image2:head {stat out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:3d3f3250:::rbd_id.image2:head sending reply 2022-03-29T17:52:03.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - with_head_obc: released 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - OpSequencer::finish_op_in_order: op=165, last_started=165, last_unblocked=165, last_completed=164 2022-03-29T17:52:03.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,805 [shard 0] osd - client_request(id=165, detail=m=[osd_op(client.4179.0:77 3.1c 3:3d3f3250:::rbd_id.image2:head {stat out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - client_request(id=166, detail=m=[osd_op(client.4179.0:78 3.1c 3.a4cfcbc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - client_request(id=166, detail=m=[osd_op(client.4179.0:78 3.1c 3.a4cfcbc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - client_request(id=166, detail=m=[osd_op(client.4179.0:78 3.1c 3.a4cfcbc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - OpSequencer::start_op: op=166, last_started=165, last_unblocked=165, last_completed=165 2022-03-29T17:52:03.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - do_recover_missing check for recovery, 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - client_request(id=166, detail=m=[osd_op(client.4179.0:78 3.1c 3:3d3f3250:::rbd_id.image2:head {call rbd.get_id in=9b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - with_head_obc 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - with_head_obc: found 3:3d3f3250:::rbd_id.image2:head in cache 2022-03-29T17:52:03.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - do_osd_ops_execute: object 3:3d3f3250:::rbd_id.image2:head - handling op call 2022-03-29T17:52:03.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - handling op call on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,806 [shard 0] osd - calling method rbd.get_id, num_read=0, num_write=0 2022-03-29T17:52:03.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - handling op stat on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - stat os.oi.size=15, os.oi.mtime=1648576322.6685143 2022-03-29T17:52:03.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - handling op sync_read on object 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - read: data length: 15 2022-03-29T17:52:03.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=15 while num_read=1, num_write=0 2022-03-29T17:52:03.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - do_osd_ops_execute: object 3:3d3f3250:::rbd_id.image2:head all operations successful 2022-03-29T17:52:03.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:78 3.1c 3:3d3f3250:::rbd_id.image2:head {call rbd.get_id in=9b out=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:3d3f3250:::rbd_id.image2:head sending reply 2022-03-29T17:52:03.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - with_head_obc: released 3:3d3f3250:::rbd_id.image2:head 2022-03-29T17:52:03.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - OpSequencer::finish_op_in_order: op=166, last_started=166, last_unblocked=166, last_completed=165 2022-03-29T17:52:03.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,807 [shard 0] osd - client_request(id=166, detail=m=[osd_op(client.4179.0:78 3.1c 3:3d3f3250:::rbd_id.image2:head {call rbd.get_id in=9b out=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4179.0:79 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4179.0:79 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4179.0:79 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.901 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - OpSequencer::start_op: op=47, last_started=46, last_unblocked=46, last_completed=46 2022-03-29T17:52:03.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4179.0:79 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_size in=19b, call rbd.get_object_prefix in=20b, call rbd.get_features in=24b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.902 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,808 [shard 0] osd - calling method rbd.get_size, num_read=0, num_write=0 2022-03-29T17:52:03.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,809 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.903 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,809 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,809 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=9 while num_read=2, num_write=0 2022-03-29T17:52:03.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,809 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,809 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,809 [shard 0] osd - calling method rbd.get_object_prefix, num_read=2, num_write=0 2022-03-29T17:52:03.904 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,810 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,810 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=24 while num_read=3, num_write=0 2022-03-29T17:52:03.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,810 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,810 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.905 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,810 [shard 0] osd - calling method rbd.get_features, num_read=3, num_write=0 2022-03-29T17:52:03.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,810 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=4, num_write=0 2022-03-29T17:52:03.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:79 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_size in=19b out=9b, call rbd.get_object_prefix in=20b out=24b, call rbd.get_features in=24b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.906 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - OpSequencer::finish_op_in_order: op=47, last_started=47, last_unblocked=47, last_completed=46 2022-03-29T17:52:03.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - client_request(id=47, detail=m=[osd_op(client.4179.0:79 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_size in=19b out=9b, call rbd.get_object_prefix in=20b out=24b, call rbd.get_features in=24b out=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4179.0:80 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.907 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4179.0:80 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4179.0:80 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,811 [shard 0] osd - OpSequencer::start_op: op=48, last_started=47, last_unblocked=47, last_completed=47 2022-03-29T17:52:03.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4179.0:80 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_create_timestamp in=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.908 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - calling method rbd.get_create_timestamp, num_read=0, num_write=0 2022-03-29T17:52:03.909 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:03.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,812 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:80 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_create_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.910 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - OpSequencer::finish_op_in_order: op=48, last_started=48, last_unblocked=48, last_completed=47 2022-03-29T17:52:03.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - client_request(id=48, detail=m=[osd_op(client.4179.0:80 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_create_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4179.0:81 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4179.0:81 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.911 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4179.0:81 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - OpSequencer::start_op: op=49, last_started=48, last_unblocked=48, last_completed=48 2022-03-29T17:52:03.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4179.0:81 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_access_timestamp in=23b, call rbd.get_modify_timestamp in=23b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.912 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,813 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,814 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,814 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,814 [shard 0] osd - calling method rbd.get_access_timestamp, num_read=0, num_write=0 2022-03-29T17:52:03.913 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,814 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,814 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:03.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,814 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,814 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,814 [shard 0] osd - calling method rbd.get_modify_timestamp, num_read=1, num_write=0 2022-03-29T17:52:03.914 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=2, num_write=0 2022-03-29T17:52:03.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:81 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_access_timestamp in=23b out=8b, call rbd.get_modify_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.915 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - OpSequencer::finish_op_in_order: op=49, last_started=49, last_unblocked=49, last_completed=48 2022-03-29T17:52:03.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - client_request(id=49, detail=m=[osd_op(client.4179.0:81 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_access_timestamp in=23b out=8b, call rbd.get_modify_timestamp in=23b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4179.0:82 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,815 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4179.0:82 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.916 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4179.0:82 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - OpSequencer::start_op: op=50, last_started=49, last_unblocked=49, last_completed=49 2022-03-29T17:52:03.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4179.0:82 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_data_pool in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.917 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - calling method rbd.get_data_pool, num_read=0, num_write=0 2022-03-29T17:52:03.918 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,816 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,817 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=1, num_write=0 2022-03-29T17:52:03.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,817 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,817 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:82 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_data_pool in=16b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.919 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,817 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,817 [shard 0] osd - OpSequencer::finish_op_in_order: op=50, last_started=50, last_unblocked=50, last_completed=49 2022-03-29T17:52:03.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,817 [shard 0] osd - client_request(id=50, detail=m=[osd_op(client.4179.0:82 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_data_pool in=16b out=8b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,817 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4179.0:83 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,817 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4179.0:83 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.920 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4179.0:83 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - OpSequencer::start_op: op=51, last_started=50, last_unblocked=50, last_completed=50 2022-03-29T17:52:03.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.921 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,818 [shard 0] alienstore - stat 2022-03-29T17:52:03.921 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4179.0:83 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_size in=19b, call rbd.get_features in=24b, call rbd.get_flags in=20b, call rbd.get_snapcontext in=18b, call lock.get_info in=30b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.922 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,818 [shard 0] osd - calling method rbd.get_size, num_read=0, num_write=0 2022-03-29T17:52:03.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,819 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,819 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,819 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=9 while num_read=2, num_write=0 2022-03-29T17:52:03.923 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,819 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,819 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,819 [shard 0] osd - calling method rbd.get_features, num_read=2, num_write=0 2022-03-29T17:52:03.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=16 while num_read=3, num_write=0 2022-03-29T17:52:03.924 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - calling method rbd.get_flags, num_read=3, num_write=0 2022-03-29T17:52:03.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=8 while num_read=4, num_write=0 2022-03-29T17:52:03.925 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,820 [shard 0] osd - calling method rbd.get_snapcontext, num_read=4, num_write=0 2022-03-29T17:52:03.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,821 [shard 0] osd - handling op omap-get-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.926 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,821 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,821 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=12 while num_read=6, num_write=0 2022-03-29T17:52:03.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,821 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,821 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,821 [shard 0] osd - calling method lock.get_info, num_read=6, num_write=0 2022-03-29T17:52:03.927 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,822 [shard 0] osd - handling op getxattr on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,822 [shard 0] osd - getxattr on obj=3:5741e4f5:::rbd_header.1053eb73922:head for attr=_lock.rbd_lock 2022-03-29T17:52:03.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,822 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=15 while num_read=7, num_write=0 2022-03-29T17:52:03.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,822 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.928 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,822 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:83 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_size in=19b out=9b, call rbd.get_features in=24b out=16b, call rbd.get_flags in=20b out=8b, call rbd.get_snapcontext in=18b out=12b, call lock.get_info in=30b out=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,822 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,822 [shard 0] osd - OpSequencer::finish_op_in_order: op=51, last_started=51, last_unblocked=51, last_completed=50 2022-03-29T17:52:03.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,822 [shard 0] osd - client_request(id=51, detail=m=[osd_op(client.4179.0:83 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.get_size in=19b out=9b, call rbd.get_features in=24b out=16b, call rbd.get_flags in=20b out=8b, call rbd.get_snapcontext in=18b out=12b, call lock.get_info in=30b out=15b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4179.0:84 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.929 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4179.0:84 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4179.0:84 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - OpSequencer::start_op: op=52, last_started=51, last_unblocked=51, last_completed=51 2022-03-29T17:52:03.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.930 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4179.0:84 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.parent_get in=13b, call rbd.parent_overlap_get in=29b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.931 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.932 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,823 [shard 0] osd - calling method rbd.parent_get, num_read=0, num_write=0 2022-03-29T17:52:03.932 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:02,824 [shard 0] alienstore - stat 2022-03-29T17:52:03.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,824 [shard 0] osd - handling op stat on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,824 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.7500558 2022-03-29T17:52:03.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,824 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.936 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,824 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,824 [shard 0] osd - handling op omap-get-vals on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,825 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=30 while num_read=3, num_write=0 2022-03-29T17:52:03.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,825 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.937 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,825 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,825 [shard 0] osd - calling method rbd.parent_overlap_get, num_read=3, num_write=0 2022-03-29T17:52:03.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,825 [shard 0] osd - handling op stat on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,825 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.7500558 2022-03-29T17:52:03.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,825 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.938 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,825 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=1 while num_read=5, num_write=0 2022-03-29T17:52:03.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:84 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.parent_get in=13b out=30b, call rbd.parent_overlap_get in=29b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.939 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - OpSequencer::finish_op_in_order: op=52, last_started=52, last_unblocked=52, last_completed=51 2022-03-29T17:52:03.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - client_request(id=52, detail=m=[osd_op(client.4179.0:84 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.parent_get in=13b out=30b, call rbd.parent_overlap_get in=29b out=1b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4179.0:85 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4179.0:85 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.940 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4179.0:85 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.941 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - OpSequencer::start_op: op=53, last_started=52, last_unblocked=52, last_completed=52 2022-03-29T17:52:03.941 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.941 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,826 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4179.0:85 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.942 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - handling op omap-get-vals on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,827 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:85 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.943 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.944 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - OpSequencer::finish_op_in_order: op=53, last_started=53, last_unblocked=53, last_completed=52 2022-03-29T17:52:03.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - client_request(id=167, detail=m=[osd_op(client.4179.0:86 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - client_request(id=167, detail=m=[osd_op(client.4179.0:86 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - client_request(id=167, detail=m=[osd_op(client.4179.0:86 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - OpSequencer::start_op: op=167, last_started=162, last_unblocked=162, last_completed=162 2022-03-29T17:52:03.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - do_recover_missing check for recovery, 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - client_request(id=167, detail=m=[osd_op(client.4179.0:86 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - with_head_obc 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - with_head_obc: found 3:5cea7035:::rbd_info:head in cache 2022-03-29T17:52:03.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head - handling op call 2022-03-29T17:52:03.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - handling op call on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - calling method rbd.metadata_list, num_read=0, num_write=0 2022-03-29T17:52:03.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,829 [shard 0] osd - handling op omap-get-vals on object 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,829 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=4 while num_read=1, num_write=0 2022-03-29T17:52:03.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,829 [shard 0] osd - do_osd_ops_execute: object 3:5cea7035:::rbd_info:head all operations successful 2022-03-29T17:52:03.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,829 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:86 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5cea7035:::rbd_info:head sending reply 2022-03-29T17:52:03.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,829 [shard 0] osd - with_head_obc: released 3:5cea7035:::rbd_info:head 2022-03-29T17:52:03.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,829 [shard 0] osd - OpSequencer::finish_op_in_order: op=167, last_started=167, last_unblocked=167, last_completed=162 2022-03-29T17:52:03.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,829 [shard 0] osd - client_request(id=167, detail=m=[osd_op(client.4179.0:86 3.1a 3:5cea7035:::rbd_info:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.954 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,828 [shard 0] osd - client_request(id=53, detail=m=[osd_op(client.4179.0:85 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.metadata_list in=33b out=4b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.954 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4179.0:87 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.954 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4179.0:87 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4179.0:87 3.a 3.af2782ea (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - OpSequencer::start_op: op=54, last_started=53, last_unblocked=53, last_completed=53 2022-03-29T17:52:03.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4179.0:87 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.image_group_get in=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.955 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.956 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,830 [shard 0] osd - calling method rbd.image_group_get, num_read=0, num_write=0 2022-03-29T17:52:03.957 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,831 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.957 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,831 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=18 while num_read=1, num_write=0 2022-03-29T17:52:03.957 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,831 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.957 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,831 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:87 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.image_group_get in=18b out=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.957 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,831 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.958 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,831 [shard 0] osd - OpSequencer::finish_op_in_order: op=54, last_started=54, last_unblocked=54, last_completed=53 2022-03-29T17:52:03.958 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,831 [shard 0] osd - client_request(id=54, detail=m=[osd_op(client.4179.0:87 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.image_group_get in=18b out=18b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.958 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4179.0:88 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.958 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4179.0:88 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.958 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4179.0:88 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - OpSequencer::start_op: op=55, last_started=54, last_unblocked=54, last_completed=54 2022-03-29T17:52:03.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4179.0:88 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {watch watch cookie 140032644115424} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.959 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op watch 2022-03-29T17:52:03.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - handling op watch on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - do_op_watch 2022-03-29T17:52:03.960 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - do_op_watch_subop_watch 2022-03-29T17:52:03.961 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,832 [shard 0] osd - registered new watch watch(cookie 140032644115424 30s 172.21.15.138:0/2702680834) by client.4179 2022-03-29T17:52:03.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head submitting txn 2022-03-29T17:52:03.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean calc_trim_to_aggressive limit = 18'1 2022-03-29T17:52:03.961 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean append_log log((0'0,18'1], crt=18'1) {18'2 (18'1) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:88 2022-03-29T17:52:02.831911+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'1 lua=0'0 crt=18'1 lcod 18'1 mlcod 18'1 active+clean add_log_entry 18'2 (18'1) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:88 2022-03-29T17:52:02.831911+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'1 lua=0'0 crt=18'2 lcod 18'1 mlcod 18'1 active+clean append_log approx pg log length = 2 2022-03-29T17:52:03.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'1 lua=0'0 crt=18'2 lcod 18'1 mlcod 18'1 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.962 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,832 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.963 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,833 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.963 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:02,834 [shard 0] alienstore - stat 2022-03-29T17:52:03.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,835 [shard 0] osd - replicated_request(id=9, detail=RepRequest(from=0 req=osd_repop(client.4179.0:88 3.a e18/14 3:5741e4f5:::rbd_header.1053eb73922:head v 18'2, mlcod=18'2) v3)) start 2022-03-29T17:52:03.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,836 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active append_log log((0'0,18'1], crt=18'1) {18'2 (18'1) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:88 2022-03-29T17:52:02.831911+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0} 2022-03-29T17:52:03.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,836 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 lcod 18'1 mlcod 18'1 active add_log_entry 18'2 (18'1) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:88 2022-03-29T17:52:02.831911+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-29T17:52:03.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,836 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'2 lcod 18'1 mlcod 18'1 active append_log approx pg log length = 2 2022-03-29T17:52:03.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,836 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'2 lcod 18'1 mlcod 18'1 active append_log transaction_applied = 1 2022-03-29T17:52:03.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,836 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,836 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'2, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,836 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.969 INFO:tasks.ceph.osd.0.smithi138.stderr:INFO 2022-03-29 17:52:02,876 [shard 0] osd - op_effect: added new watcher: 140032644115424,client.4179 2022-03-29T17:52:03.970 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,876 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:88 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {watch watch cookie 140032644115424} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.970 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.970 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - OpSequencer::finish_op_in_order: op=55, last_started=55, last_unblocked=55, last_completed=54 2022-03-29T17:52:03.970 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - client_request(id=55, detail=m=[osd_op(client.4179.0:88 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {watch watch cookie 140032644115424} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.971 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4179.0:89 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.971 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4179.0:89 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.971 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4179.0:89 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.971 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - OpSequencer::start_op: op=56, last_started=55, last_unblocked=55, last_completed=55 2022-03-29T17:52:03.971 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.972 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4179.0:89 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.parent_attach in=67b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.972 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,877 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.972 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - with_head_obc: found 3:5741e4f5:::rbd_header.1053eb73922:head in cache 2022-03-29T17:52:03.972 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head - handling op call 2022-03-29T17:52:03.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - handling op call on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - calling method rbd.parent_attach, num_read=0, num_write=0 2022-03-29T17:52:03.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - handling op stat on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - stat os.oi.size=0, os.oi.mtime=1648576322.831912 2022-03-29T17:52:03.973 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,878 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - handling op omap-set-vals on object 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=0 while num_read=3, num_write=1 2022-03-29T17:52:03.974 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head all operations successful 2022-03-29T17:52:03.975 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - do_osd_ops_execute: object 3:5741e4f5:::rbd_header.1053eb73922:head submitting txn 2022-03-29T17:52:03.975 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=18'2 mlcod 18'2 active+clean calc_trim_to_aggressive limit = 18'2 2022-03-29T17:52:03.975 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 lua=0'0 crt=18'2 mlcod 18'2 active+clean append_log log((0'0,18'2], crt=18'2) {18'3 (18'2) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:89 2022-03-29T17:52:02.877402+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.975 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'2 lua=0'0 crt=18'2 lcod 18'2 mlcod 18'2 active+clean add_log_entry 18'3 (18'2) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:89 2022-03-29T17:52:02.877402+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'2 lua=0'0 crt=18'3 lcod 18'2 mlcod 18'2 active+clean append_log approx pg log length = 3 2022-03-29T17:52:03.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=0 lpr=14 luod=18'2 lua=0'0 crt=18'3 lcod 18'2 mlcod 18'2 active+clean append_log transaction_applied = 1 2022-03-29T17:52:03.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.976 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,879 [shard 0] osd - ReplicatedBackend::_submit_transaction: do_transaction... 2022-03-29T17:52:03.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,917 [shard 0] osd - replicated_request(id=10, detail=RepRequest(from=0 req=osd_repop(client.4179.0:89 3.a e18/14 3:5741e4f5:::rbd_header.1053eb73922:head v 18'3, mlcod=18'3) v3)) start 2022-03-29T17:52:03.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,917 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'2 (0'0,18'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'2 mlcod 18'2 active append_log log((0'0,18'2], crt=18'2) {18'3 (18'2) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:89 2022-03-29T17:52:02.877402+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0} 2022-03-29T17:52:03.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,917 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'2 lcod 18'2 mlcod 18'2 active add_log_entry 18'3 (18'2) modify 3:5741e4f5:::rbd_header.1053eb73922:head by client.4179.0:89 2022-03-29T17:52:02.877402+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 0, new_object: 0 2022-03-29T17:52:03.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,917 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 lcod 18'2 mlcod 18'2 active append_log approx pg log length = 3 2022-03-29T17:52:03.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,917 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 lcod 18'2 mlcod 18'2 active append_log transaction_applied = 1 2022-03-29T17:52:03.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,917 [shard 0] osd - trim proposed trim_to = 0'0 2022-03-29T17:52:03.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,917 [shard 0] osd - write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 18'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0 2022-03-29T17:52:03.988 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:02,918 [shard 0] osd - PG::handle_rep_op: do_transaction... 2022-03-29T17:52:03.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,958 [shard 0] osd - do_osd_ops: osd_op(client.4179.0:89 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.parent_attach in=67b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8 - object 3:5741e4f5:::rbd_header.1053eb73922:head sending reply 2022-03-29T17:52:03.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - with_head_obc: released 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:03.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - OpSequencer::finish_op_in_order: op=56, last_started=56, last_unblocked=56, last_completed=55 2022-03-29T17:52:03.993 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - client_request(id=56, detail=m=[osd_op(client.4179.0:89 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {call rbd.parent_attach in=67b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:03.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4179.0:90 3.a 3.fa1caca (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:03.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4179.0:90 3.a 3.fa1caca (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:03.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4179.0:90 3.a 3.fa1caca (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:03.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - OpSequencer::start_op: op=57, last_started=56, last_unblocked=56, last_completed=56 2022-03-29T17:52:03.994 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - do_recover_missing check for recovery, 3:535385f0:::rbd_children:head 2022-03-29T17:52:03.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - client_request(id=57, detail=m=[osd_op(client.4179.0:90 3.a 3:535385f0:::rbd_children:head {call rbd.add_child in=59b} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:03.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - with_head_obc 3:535385f0:::rbd_children:head 2022-03-29T17:52:03.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,959 [shard 0] osd - with_head_obc: cache miss on 3:535385f0:::rbd_children:head 2022-03-29T17:52:03.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,960 [shard 0] osd - load_metadata: object 3:535385f0:::rbd_children:head doesn't exist, returning empty metadata 2022-03-29T17:52:03.995 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,960 [shard 0] osd - load_head_obc: loaded obs 3:535385f0:::rbd_children:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:535385f0:::rbd_children:head 2022-03-29T17:52:03.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,960 [shard 0] osd - load_head_obc: returning obc 3:535385f0:::rbd_children:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:535385f0:::rbd_children:head 2022-03-29T17:52:03.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,960 [shard 0] osd - do_osd_ops_execute: object 3:535385f0:::rbd_children:head - handling op call 2022-03-29T17:52:03.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,960 [shard 0] osd - handling op call on object 3:535385f0:::rbd_children:head 2022-03-29T17:52:03.996 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,960 [shard 0] osd - calling method rbd.add_child, num_read=0, num_write=0 2022-03-29T17:52:03.997 INFO:tasks.ceph.osd.0.smithi138.stderr:DEBUG 2022-03-29 17:52:02,960 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:535385f0:::rbd_children:head 2022-03-29T17:52:03.997 INFO:tasks.ceph.osd.0.smithi138.stderr:================================================================= 2022-03-29T17:52:03.997 INFO:tasks.ceph.osd.0.smithi138.stderr:==35577==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fb5e50f5308 at pc 0x5601356c90f1 bp 0x7fb5e50f52d0 sp 0x7fb5e50f52c0 2022-03-29T17:52:03.997 INFO:tasks.ceph.osd.0.smithi138.stderr:READ of size 8 at 0x7fb5e50f5308 thread T0 2022-03-29T17:52:03.997 INFO:tasks.ceph.osd.0.smithi138.stderr: #0 0x5601356c90f0 (/usr/bin/ceph-osd+0x478d20f0) 2022-03-29T17:52:03.998 INFO:tasks.ceph.osd.0.smithi138.stderr: #1 0x5601356c8063 in dl_iterate_phdr (/usr/bin/ceph-osd+0x478d1063) 2022-03-29T17:52:03.998 INFO:tasks.ceph.osd.0.smithi138.stderr: #2 0x7fb613769174 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13174) 2022-03-29T17:52:03.998 INFO:tasks.ceph.osd.0.smithi138.stderr: #3 0x7fb613765712 (/lib64/libgcc_s.so.1+0xf712) 2022-03-29T17:52:03.998 INFO:tasks.ceph.osd.0.smithi138.stderr: #4 0x7fb6137668ef (/lib64/libgcc_s.so.1+0x108ef) 2022-03-29T17:52:03.999 INFO:tasks.ceph.osd.0.smithi138.stderr: #5 0x7fb61376732a in _Unwind_Resume (/lib64/libgcc_s.so.1+0x1132a) 2022-03-29T17:52:03.999 INFO:tasks.ceph.osd.0.smithi138.stderr: #6 0x560128953283 (/usr/bin/ceph-osd+0x3ab5c283) 2022-03-29T17:52:03.999 INFO:tasks.ceph.osd.0.smithi138.stderr: #7 0x560128954f33 in seastar::logger::lambda_log_writer(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x3ab5df33) 2022-03-29T17:52:03.999 INFO:tasks.ceph.osd.0.smithi138.stderr: #8 0x560136cf4f81 (/usr/bin/ceph-osd+0x48efdf81) 2022-03-29T17:52:03.999 INFO:tasks.ceph.osd.0.smithi138.stderr: #9 0x560129d754fa in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const (/usr/bin/ceph-osd+0x3bf7e4fa) 2022-03-29T17:52:04.000 INFO:tasks.ceph.osd.0.smithi138.stderr: #10 0x560129fe421c (/usr/bin/ceph-osd+0x3c1ed21c) 2022-03-29T17:52:04.000 INFO:tasks.ceph.osd.0.smithi138.stderr: #11 0x56012a18a8b0 in crimson::osd::OpsExecuter::execute_op(OSDOp&) (/usr/bin/ceph-osd+0x3c3938b0) 2022-03-29T17:52:04.000 INFO:tasks.ceph.osd.0.smithi138.stderr: #12 0x56012b195310 (/usr/bin/ceph-osd+0x3d39e310) 2022-03-29T17:52:04.000 INFO:tasks.ceph.osd.0.smithi138.stderr: #13 0x56012b1ef45b in cls_cxx_map_get_val(void*, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list*) (/usr/bin/ceph-osd+0x3d3f845b) 2022-03-29T17:52:04.000 INFO:tasks.ceph.osd.0.smithi138.stderr: #14 0x7fb5ff4585c0 (/usr/lib64/rados-classes/libcls_rbd.so.1.0.0+0x3e5c0) 2022-03-29T17:52:04.001 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:52:04.001 INFO:tasks.ceph.osd.0.smithi138.stderr:0x7fb5e50f5308 is located 242440 bytes inside of 262144-byte region [0x7fb5e50ba000,0x7fb5e50fa000) 2022-03-29T17:52:04.001 INFO:tasks.ceph.osd.0.smithi138.stderr:allocated by thread T0 here: 2022-03-29T17:52:04.001 INFO:tasks.ceph.osd.0.smithi138.stderr: #0 0x7fb617c79780 in __interceptor_aligned_alloc (/lib64/libasan.so.5+0xf0780) 2022-03-29T17:52:04.002 INFO:tasks.ceph.osd.0.smithi138.stderr: #1 0x5601360541c1 (/usr/bin/ceph-osd+0x4825d1c1) 2022-03-29T17:52:04.002 INFO:tasks.ceph.osd.0.smithi138.stderr: #2 0x7ffc1274e40f ([stack]+0xa440f) 2022-03-29T17:52:04.002 INFO:tasks.ceph.osd.0.smithi138.stderr: 2022-03-29T17:52:04.002 INFO:tasks.ceph.osd.0.smithi138.stderr:SUMMARY: AddressSanitizer: stack-buffer-overflow (/usr/bin/ceph-osd+0x478d20f0) 2022-03-29T17:52:04.002 INFO:tasks.ceph.osd.0.smithi138.stderr:Shadow bytes around the buggy address: 2022-03-29T17:52:04.003 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16a10: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:04.003 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16a20: f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:04.003 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16a30: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:04.003 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16a40: f8 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 00 00 00 2022-03-29T17:52:04.003 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16a50: 00 00 00 00 00 00 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:04.004 INFO:tasks.ceph.osd.0.smithi138.stderr:=>0x0ff73ca16a60: 00[f2]f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:04.004 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16a70: 00 f2 f2 f2 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 2022-03-29T17:52:04.004 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16a80: 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 2022-03-29T17:52:04.004 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16a90: 00 00 f2 f2 00 00 00 f2 f2 f2 f2 f2 00 00 00 00 2022-03-29T17:52:04.004 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16aa0: f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00 00 00 2022-03-29T17:52:04.005 INFO:tasks.ceph.osd.0.smithi138.stderr: 0x0ff73ca16ab0: f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00 00 00 2022-03-29T17:52:04.005 INFO:tasks.ceph.osd.0.smithi138.stderr:Shadow byte legend (one shadow byte represents 8 application bytes): 2022-03-29T17:52:04.005 INFO:tasks.ceph.osd.0.smithi138.stderr: Addressable: 00 2022-03-29T17:52:04.005 INFO:tasks.ceph.osd.0.smithi138.stderr: Partially addressable: 01 02 03 04 05 06 07 2022-03-29T17:52:04.006 INFO:tasks.ceph.osd.0.smithi138.stderr: Heap left redzone: fa 2022-03-29T17:52:04.006 INFO:tasks.ceph.osd.0.smithi138.stderr: Freed heap region: fd 2022-03-29T17:52:04.006 INFO:tasks.ceph.osd.0.smithi138.stderr: Stack left redzone: f1 2022-03-29T17:52:04.006 INFO:tasks.ceph.osd.0.smithi138.stderr: Stack mid redzone: f2 2022-03-29T17:52:04.006 INFO:tasks.ceph.osd.0.smithi138.stderr: Stack right redzone: f3 2022-03-29T17:52:04.007 INFO:tasks.ceph.osd.0.smithi138.stderr: Stack after return: f5 2022-03-29T17:52:04.007 INFO:tasks.ceph.osd.0.smithi138.stderr: Stack use after scope: f8 2022-03-29T17:52:04.007 INFO:tasks.ceph.osd.0.smithi138.stderr: Global redzone: f9 2022-03-29T17:52:04.007 INFO:tasks.ceph.osd.0.smithi138.stderr: Global init order: f6 2022-03-29T17:52:04.007 INFO:tasks.ceph.osd.0.smithi138.stderr: Poisoned by user: f7 2022-03-29T17:52:04.008 INFO:tasks.ceph.osd.0.smithi138.stderr: Container overflow: fc 2022-03-29T17:52:04.008 INFO:tasks.ceph.osd.0.smithi138.stderr: Array cookie: ac 2022-03-29T17:52:04.008 INFO:tasks.ceph.osd.0.smithi138.stderr: Intra object redzone: bb 2022-03-29T17:52:04.008 INFO:tasks.ceph.osd.0.smithi138.stderr: ASan internal: fe 2022-03-29T17:52:04.009 INFO:tasks.ceph.osd.0.smithi138.stderr: Left alloca redzone: ca 2022-03-29T17:52:04.009 INFO:tasks.ceph.osd.0.smithi138.stderr: Right alloca redzone: cb 2022-03-29T17:52:04.009 INFO:tasks.ceph.osd.0.smithi138.stderr:==35577==ABORTING 2022-03-29T17:52:04.010 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,093 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577@61441] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:04.010 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,093 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577@61441] closing: reset yes, replace no 2022-03-29T17:52:04.010 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,093 [shard 0] osd - Heartbeat::Peer: osd.0 disconnected 2022-03-29T17:52:04.011 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,094 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578@61768 >> osd.0 v2:172.21.15.138:6804/35577] execute_ready(): fault at READY with nothing to send, going to STANDBY -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:04.011 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,094 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577@55215] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:04.012 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,094 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577@55215] closing: reset yes, replace no 2022-03-29T17:52:04.012 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,094 [shard 0] osd - Heartbeat::Peer: osd.0 disconnected 2022-03-29T17:52:04.012 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,094 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=6264692341996179416 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:04.012 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,094 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578@59901 >> osd.0 v2:172.21.15.138:6809/35577] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:04.013 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,094 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578@59901 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset yes, replace no 2022-03-29T17:52:04.013 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,094 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=5242729563455272179 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:04.014 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,093 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=17440947221162492340 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:04.014 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,095 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577@54393] execute_ready(): fault at READY with nothing to send, going to STANDBY -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:04.014 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,095 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577@52708] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:04.014 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,095 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577@52708] closing: reset yes, replace no 2022-03-29T17:52:04.015 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,095 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=913655188780050447 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:04.015 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,095 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.016 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:03,095 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:04.016 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,095 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.016 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:03,095 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:04.017 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,096 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.017 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:03,096 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:04.018 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,096 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.018 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:03,096 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:04.019 INFO:tasks.ceph.osd.0.smithi138.stderr:daemon-helper: command failed with exit status 1 2022-03-29T17:52:04.019 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,295 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.020 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,295 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.020 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,296 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.021 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,296 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.021 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,296 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.022 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:03,296 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:04.022 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,296 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.022 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:03,297 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:04.023 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,296 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.023 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:03,297 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:04.023 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,297 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.024 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:03,297 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:04.024 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,697 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.025 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,697 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.025 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,697 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.025 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,697 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.026 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,697 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.026 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,698 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.027 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:03,698 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:04.027 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,698 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.027 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:03,698 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:04.028 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:03,698 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:04.028 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,698 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.028 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:03,698 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:04.029 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:03,832 [shard 0] alienstore - stat 2022-03-29T17:52:04.029 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:03,842 [shard 0] alienstore - stat 2022-03-29T17:52:04.500 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:04,498 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.500 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:04,498 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.501 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:04,499 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.501 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:04,499 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:04.502 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:04,499 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.502 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:04,499 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:04.503 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:04,499 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.503 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:04,499 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:04.504 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:04,500 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.504 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:04,500 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:04.504 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:04,500 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:04.505 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:04,500 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:04.840 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:04,838 [shard 0] alienstore - stat 2022-03-29T17:52:04.851 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:04,849 [shard 0] alienstore - stat 2022-03-29T17:52:05.847 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:05,846 [shard 0] alienstore - stat 2022-03-29T17:52:05.857 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:05,856 [shard 0] alienstore - stat 2022-03-29T17:52:06.101 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,099 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:06.101 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,099 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:06.101 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:06,099 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 3.2 seconds ... 2022-03-29T17:52:06.102 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,099 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:06.102 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,100 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:06.102 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:06,100 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 3.2 seconds ... 2022-03-29T17:52:06.103 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:06,100 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:06.104 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:06,100 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:06.104 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:06,101 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:06.104 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:06,101 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 3.2 seconds ... 2022-03-29T17:52:06.104 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:06,101 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:06.105 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:06,101 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 3.2 seconds ... 2022-03-29T17:52:06.300 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,298 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is pending send... 2022-03-29T17:52:06.350 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:06,348 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is pending send... 2022-03-29T17:52:06.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,842 [shard 0] osd - peering_event(id=487, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:52:06.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,842 [shard 0] osd - peering_event(id=487, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:52:06.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,842 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:52:06.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,843 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 141.530502319s 2022-03-29T17:52:06.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,843 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 141.530502319s (sent 149.538833618s) 2022-03-29T17:52:06.847 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,843 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578@61768 >> osd.0 v2:172.21.15.138:6804/35577] notify_write(): at STANDBY, going to CONNECTING 2022-03-29T17:52:06.847 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,843 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578@61768 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:06.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,843 [shard 0] osd - log is not dirty 2022-03-29T17:52:06.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,843 [shard 0] osd - peering_event(id=487, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:52:06.848 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,843 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:06.848 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,843 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:06.848 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:06,844 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] waiting 0.2 seconds ... 2022-03-29T17:52:06.848 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,844 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:06.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:06,844 [shard 0] osd - peering_event(id=487, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:52:06.854 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:06,852 [shard 0] alienstore - stat 2022-03-29T17:52:06.864 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:06,862 [shard 0] alienstore - stat 2022-03-29T17:52:07.046 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,044 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:07.047 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,044 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:07.047 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,044 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:07.047 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:07,044 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] waiting 0.4 seconds ... 2022-03-29T17:52:07.047 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,044 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:07.446 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,444 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:07.447 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,444 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:07.447 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,445 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:07.447 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:07,445 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] waiting 0.8 seconds ... 2022-03-29T17:52:07.447 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,445 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:07.462 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,456 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,456 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,456 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1d 2022-03-29T17:52:07.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,456 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.095642090s 2022-03-29T17:52:07.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,456 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.095642090s (sent 150.089828491s) 2022-03-29T17:52:07.464 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,456 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577@54393] notify_write(): at STANDBY, going to CONNECTING 2022-03-29T17:52:07.465 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,457 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577@54393] write_event: delay ... 2022-03-29T17:52:07.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,457 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,457 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,457 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.466 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,457 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:07.467 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:07,457 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 0.2 seconds ... 2022-03-29T17:52:07.467 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,457 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:07.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,457 [shard 0] osd - peering_event(id=223, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.479 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,476 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.479 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,476 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,476 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:52:07.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,477 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.110671997s 2022-03-29T17:52:07.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,477 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.110671997s (sent 150.110015869s) 2022-03-29T17:52:07.480 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,477 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,477 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,477 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.481 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,478 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:52:07.482 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,478 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.110671997s ub 150.110015869s int 16.000000000s))): start 2022-03-29T17:52:07.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,478 [shard 0] osd - peering_event(id=224, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.110671997s ub 150.110015869s int 16.000000000s))): got map 18 2022-03-29T17:52:07.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.110671997s ub 150.110015869s int 16.000000000s))): pg present 2022-03-29T17:52:07.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.110671997s ub 150.110015869s int 16.000000000s) for pg: 3.8 2022-03-29T17:52:07.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.110671997s ub 150.110015869s int 16.000000000s) 2022-03-29T17:52:07.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.175369263s 2022-03-29T17:52:07.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.175369263s 2022-03-29T17:52:07.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.110671997s ub 150.110015869s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,479 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,480 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.110671997s ub 150.110015869s int 16.000000000s))): complete 2022-03-29T17:52:07.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,486 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,486 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,486 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,486 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.18 2022-03-29T17:52:07.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.121215820s 2022-03-29T17:52:07.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.121215820s (sent 150.120101929s) 2022-03-29T17:52:07.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:52:07.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 142.129089355s 2022-03-29T17:52:07.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 142.129089355s (sent 150.120346069s) 2022-03-29T17:52:07.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,487 [shard 0] osd - peering_event(id=225, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,488 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:52:07.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,488 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.129089355s ub 150.120346069s int 16.000000000s))): start 2022-03-29T17:52:07.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,488 [shard 0] osd - handle_peering_op on 3.8 from 1 2022-03-29T17:52:07.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,488 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.110015869s))): start 2022-03-29T17:52:07.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,488 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.110015869s))): got map 18 2022-03-29T17:52:07.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,488 [shard 0] osd - peering_event(id=226, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.110015869s))): pg present 2022-03-29T17:52:07.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,488 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.129089355s ub 150.120346069s int 16.000000000s))): got map 18 2022-03-29T17:52:07.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.129089355s ub 150.120346069s int 16.000000000s))): pg present 2022-03-29T17:52:07.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.129089355s ub 150.120346069s int 16.000000000s) for pg: 3.7 2022-03-29T17:52:07.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active proc_lease pg_lease(ru 142.129089355s ub 150.120346069s int 16.000000000s) 2022-03-29T17:52:07.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active no peer_clock_delta_lb -> ruub 150.185317993s 2022-03-29T17:52:07.499 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active proc_lease readable_until_ub now 150.185317993s 2022-03-29T17:52:07.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.110015869s) for pg: 3.8 2022-03-29T17:52:07.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.110015869s 2022-03-29T17:52:07.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.110015869s (sent 150.110015869s) 2022-03-29T17:52:07.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.110015869s))): submitting ctx 2022-03-29T17:52:07.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,489 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.502 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,490 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.502 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,490 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.129089355s ub 150.120346069s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.503 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,490 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,490 [shard 0] osd - peering_event(id=227, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.110015869s))): complete 2022-03-29T17:52:07.503 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,491 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.129089355s ub 150.120346069s int 16.000000000s))): complete 2022-03-29T17:52:07.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,516 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,516 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,516 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:52:07.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,516 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.150024414s 2022-03-29T17:52:07.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,516 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.150024414s (sent 150.149948120s) 2022-03-29T17:52:07.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,517 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,517 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,517 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,517 [shard 0] osd - handle_peering_op on 3.7 from 1 2022-03-29T17:52:07.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,517 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.120346069s))): start 2022-03-29T17:52:07.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,517 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:52:07.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,517 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.150024414s ub 150.149948120s int 16.000000000s))): start 2022-03-29T17:52:07.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.150024414s ub 150.149948120s int 16.000000000s))): got map 18 2022-03-29T17:52:07.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.150024414s ub 150.149948120s int 16.000000000s))): pg present 2022-03-29T17:52:07.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.150024414s ub 150.149948120s int 16.000000000s) for pg: 3.1e 2022-03-29T17:52:07.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.150024414s ub 150.149948120s int 16.000000000s) 2022-03-29T17:52:07.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.214340210s 2022-03-29T17:52:07.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.214340210s 2022-03-29T17:52:07.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.150024414s ub 150.149948120s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - peering_event(id=228, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,518 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.120346069s))): got map 18 2022-03-29T17:52:07.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,519 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.120346069s))): pg present 2022-03-29T17:52:07.526 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,519 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.150024414s ub 150.149948120s int 16.000000000s))): complete 2022-03-29T17:52:07.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,519 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.120346069s) for pg: 3.7 2022-03-29T17:52:07.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,519 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 150.120346069s 2022-03-29T17:52:07.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,519 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 150.120346069s (sent 150.120346069s) 2022-03-29T17:52:07.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,519 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,519 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.120346069s))): submitting ctx 2022-03-29T17:52:07.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,519 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,520 [shard 0] osd - handle_peering_op on 3.1e from 1 2022-03-29T17:52:07.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,520 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.149948120s))): start 2022-03-29T17:52:07.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,520 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.149948120s))): got map 18 2022-03-29T17:52:07.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,520 [shard 0] osd - peering_event(id=229, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.120346069s))): complete 2022-03-29T17:52:07.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,521 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.149948120s))): pg present 2022-03-29T17:52:07.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,521 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.149948120s) for pg: 3.1e 2022-03-29T17:52:07.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,521 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.149948120s 2022-03-29T17:52:07.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,521 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.149948120s (sent 150.149948120s) 2022-03-29T17:52:07.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,521 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,521 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.149948120s))): submitting ctx 2022-03-29T17:52:07.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,521 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,521 [shard 0] osd - peering_event(id=230, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.149948120s))): complete 2022-03-29T17:52:07.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,536 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,537 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,537 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.e 2022-03-29T17:52:07.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,537 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.0 ruub 142.164947510s 2022-03-29T17:52:07.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,537 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 142.164947510s (sent 150.170211792s) 2022-03-29T17:52:07.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,537 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,537 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,537 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,537 [shard 0] osd - peering_event(id=231, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,546 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,547 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,547 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:52:07.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,547 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.175064087s 2022-03-29T17:52:07.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,547 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.175064087s (sent 150.180313110s) 2022-03-29T17:52:07.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,547 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,547 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,547 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,548 [shard 0] osd - peering_event(id=232, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,549 [shard 0] osd - peering_event(id=491, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,549 [shard 0] osd - peering_event(id=491, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,549 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.19 2022-03-29T17:52:07.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,549 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.247009277s 2022-03-29T17:52:07.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,549 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.247009277s (sent 150.245559692s) 2022-03-29T17:52:07.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,549 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,549 [shard 0] osd - peering_event(id=491, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,549 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:52:07.555 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.175064087s ub 150.180313110s int 16.000000000s))): start 2022-03-29T17:52:07.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - peering_event(id=491, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.175064087s ub 150.180313110s int 16.000000000s))): got map 18 2022-03-29T17:52:07.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.175064087s ub 150.180313110s int 16.000000000s))): pg present 2022-03-29T17:52:07.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.175064087s ub 150.180313110s int 16.000000000s) for pg: 3.16 2022-03-29T17:52:07.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.175064087s ub 150.180313110s int 16.000000000s) 2022-03-29T17:52:07.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.246734619s 2022-03-29T17:52:07.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.246734619s 2022-03-29T17:52:07.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.175064087s ub 150.180313110s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,550 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,551 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:52:07.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,551 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.247009277s ub 150.245559692s int 16.000000000s))): start 2022-03-29T17:52:07.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,551 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.247009277s ub 150.245559692s int 16.000000000s))): got map 18 2022-03-29T17:52:07.559 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.175064087s ub 150.180313110s int 16.000000000s))): complete 2022-03-29T17:52:07.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.247009277s ub 150.245559692s int 16.000000000s))): pg present 2022-03-29T17:52:07.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.247009277s ub 150.245559692s int 16.000000000s) for pg: 3.19 2022-03-29T17:52:07.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.247009277s ub 150.245559692s int 16.000000000s) 2022-03-29T17:52:07.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.185775757s 2022-03-29T17:52:07.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.185775757s 2022-03-29T17:52:07.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.247009277s ub 150.245559692s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,552 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,553 [shard 0] osd - handle_peering_op on 3.16 from 1 2022-03-29T17:52:07.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,553 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.180313110s))): start 2022-03-29T17:52:07.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,553 [shard 0] osd - handle_peering_op on 3.19 from 2 2022-03-29T17:52:07.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,553 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.245559692s))): start 2022-03-29T17:52:07.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,553 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.245559692s))): got map 18 2022-03-29T17:52:07.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,553 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.245559692s))): pg present 2022-03-29T17:52:07.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,554 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.245559692s) for pg: 3.19 2022-03-29T17:52:07.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,554 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.245559692s 2022-03-29T17:52:07.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,554 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.245559692s (sent 150.245559692s) 2022-03-29T17:52:07.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,554 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,554 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.245559692s))): submitting ctx 2022-03-29T17:52:07.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,554 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.245559692s))): complete 2022-03-29T17:52:07.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,554 [shard 0] osd - peering_event(id=233, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.247009277s ub 150.245559692s int 16.000000000s))): complete 2022-03-29T17:52:07.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.180313110s))): got map 18 2022-03-29T17:52:07.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.180313110s))): pg present 2022-03-29T17:52:07.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.180313110s) for pg: 3.16 2022-03-29T17:52:07.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.180313110s 2022-03-29T17:52:07.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.180313110s (sent 150.180313110s) 2022-03-29T17:52:07.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.180313110s))): submitting ctx 2022-03-29T17:52:07.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,555 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,556 [shard 0] osd - peering_event(id=234, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.180313110s))): complete 2022-03-29T17:52:07.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,556 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,556 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,557 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:52:07.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,557 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.193099976s 2022-03-29T17:52:07.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,557 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.193099976s (sent 150.190109253s) 2022-03-29T17:52:07.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,557 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,557 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,557 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,559 [shard 0] osd - peering_event(id=235, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,587 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,587 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,587 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:52:07.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,587 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 142.221298218s 2022-03-29T17:52:07.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,587 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.221298218s (sent 150.220336914s) 2022-03-29T17:52:07.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,587 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,587 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,587 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,588 [shard 0] osd - peering_event(id=236, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,595 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:52:07.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,595 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.193099976s ub 150.190109253s int 16.000000000s))): start 2022-03-29T17:52:07.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,595 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.193099976s ub 150.190109253s int 16.000000000s))): got map 18 2022-03-29T17:52:07.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,595 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.193099976s ub 150.190109253s int 16.000000000s))): pg present 2022-03-29T17:52:07.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,595 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.193099976s ub 150.190109253s int 16.000000000s) for pg: 3.5 2022-03-29T17:52:07.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.193099976s ub 150.190109253s int 16.000000000s) 2022-03-29T17:52:07.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.291976929s 2022-03-29T17:52:07.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.291976929s 2022-03-29T17:52:07.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.193099976s ub 150.190109253s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:52:07.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.221298218s ub 150.220336914s int 16.000000000s))): start 2022-03-29T17:52:07.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.221298218s ub 150.220336914s int 16.000000000s))): got map 18 2022-03-29T17:52:07.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.221298218s ub 150.220336914s int 16.000000000s))): pg present 2022-03-29T17:52:07.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.193099976s ub 150.190109253s int 16.000000000s))): complete 2022-03-29T17:52:07.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,596 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.221298218s ub 150.220336914s int 16.000000000s) for pg: 3.11 2022-03-29T17:52:07.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,597 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.221298218s ub 150.220336914s int 16.000000000s) 2022-03-29T17:52:07.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,597 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.292968750s 2022-03-29T17:52:07.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,597 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.292968750s 2022-03-29T17:52:07.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,597 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,597 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.221298218s ub 150.220336914s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,597 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,599 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 142.221298218s ub 150.220336914s int 16.000000000s))): complete 2022-03-29T17:52:07.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,599 [shard 0] osd - peering_event(id=496, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,599 [shard 0] osd - peering_event(id=496, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,599 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.14 2022-03-29T17:52:07.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,599 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.294967651s 2022-03-29T17:52:07.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,600 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.294967651s (sent 150.295852661s) 2022-03-29T17:52:07.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,600 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,600 [shard 0] osd - peering_event(id=496, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,600 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,600 [shard 0] osd - peering_event(id=496, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,619 [shard 0] osd - peering_event(id=497, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,619 [shard 0] osd - peering_event(id=498, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,619 [shard 0] osd - peering_event(id=498, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.623 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - peering_event(id=497, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1c 2022-03-29T17:52:07.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean recalc_readable_until peer osd.0 ruub 142.314071655s 2022-03-29T17:52:07.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean recalc_readable_until readable_until[_ub] 142.314071655s (sent 150.316085815s) 2022-03-29T17:52:07.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - peering_event(id=497, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.0 2022-03-29T17:52:07.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.317977905s 2022-03-29T17:52:07.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.317977905s (sent 150.316345215s) 2022-03-29T17:52:07.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - peering_event(id=498, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,620 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,621 [shard 0] osd - peering_event(id=497, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,621 [shard 0] osd - peering_event(id=498, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,636 [shard 0] osd - handle_peering_op on 3.5 from 1 2022-03-29T17:52:07.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,636 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.190109253s))): start 2022-03-29T17:52:07.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,637 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.190109253s))): got map 18 2022-03-29T17:52:07.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,637 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.190109253s))): pg present 2022-03-29T17:52:07.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,637 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.190109253s) for pg: 3.5 2022-03-29T17:52:07.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,637 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.190109253s 2022-03-29T17:52:07.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,637 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.190109253s (sent 150.190109253s) 2022-03-29T17:52:07.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,637 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,637 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.190109253s))): submitting ctx 2022-03-29T17:52:07.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,637 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - handle_peering_op on 3.11 from 1 2022-03-29T17:52:07.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.220336914s))): start 2022-03-29T17:52:07.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.220336914s))): got map 18 2022-03-29T17:52:07.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.220336914s))): pg present 2022-03-29T17:52:07.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - peering_event(id=237, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.190109253s))): complete 2022-03-29T17:52:07.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:52:07.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.294967651s ub 150.295852661s int 16.000000000s))): start 2022-03-29T17:52:07.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.220336914s) for pg: 3.11 2022-03-29T17:52:07.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.220336914s 2022-03-29T17:52:07.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.220336914s (sent 150.220336914s) 2022-03-29T17:52:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,638 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.220336914s))): submitting ctx 2022-03-29T17:52:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.294967651s ub 150.295852661s int 16.000000000s))): got map 18 2022-03-29T17:52:07.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.294967651s ub 150.295852661s int 16.000000000s))): pg present 2022-03-29T17:52:07.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.294967651s ub 150.295852661s int 16.000000000s) for pg: 3.14 2022-03-29T17:52:07.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.294967651s ub 150.295852661s int 16.000000000s) 2022-03-29T17:52:07.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.272644043s 2022-03-29T17:52:07.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.272644043s 2022-03-29T17:52:07.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.294967651s ub 150.295852661s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,639 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:52:07.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.317977905s ub 150.316345215s int 16.000000000s))): start 2022-03-29T17:52:07.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.317977905s ub 150.316345215s int 16.000000000s))): got map 18 2022-03-29T17:52:07.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.317977905s ub 150.316345215s int 16.000000000s))): pg present 2022-03-29T17:52:07.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - peering_event(id=499, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - peering_event(id=500, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - peering_event(id=499, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - peering_event(id=500, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1a 2022-03-29T17:52:07.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until peer osd.2 ruub 142.332717896s 2022-03-29T17:52:07.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until readable_until[_ub] 142.332717896s (sent 150.336608887s) 2022-03-29T17:52:07.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - peering_event(id=499, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,640 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,641 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.b 2022-03-29T17:52:07.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,641 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.336990356s 2022-03-29T17:52:07.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,641 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.336990356s (sent 150.336990356s) 2022-03-29T17:52:07.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,641 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,641 [shard 0] osd - peering_event(id=500, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,641 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,641 [shard 0] osd - handle_peering_op on 3.14 from 2 2022-03-29T17:52:07.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,641 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.295852661s))): start 2022-03-29T17:52:07.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,642 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.295852661s))): got map 18 2022-03-29T17:52:07.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,642 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.295852661s))): pg present 2022-03-29T17:52:07.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,642 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.295852661s) for pg: 3.14 2022-03-29T17:52:07.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,642 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.295852661s 2022-03-29T17:52:07.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,642 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.295852661s (sent 150.295852661s) 2022-03-29T17:52:07.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,642 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,642 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.295852661s))): submitting ctx 2022-03-29T17:52:07.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,642 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - peering_event(id=238, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 150.220336914s))): complete 2022-03-29T17:52:07.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.317977905s ub 150.316345215s int 16.000000000s) for pg: 3.0 2022-03-29T17:52:07.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.317977905s ub 150.316345215s int 16.000000000s) 2022-03-29T17:52:07.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.276321411s 2022-03-29T17:52:07.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.276321411s 2022-03-29T17:52:07.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.317977905s ub 150.316345215s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,643 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:52:07.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - peering_event(id=499, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - peering_event(id=500, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.295852661s))): complete 2022-03-29T17:52:07.664 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - handle_peering_op on 3.0 from 2 2022-03-29T17:52:07.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.316345215s))): start 2022-03-29T17:52:07.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.316345215s))): got map 18 2022-03-29T17:52:07.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.316345215s))): pg present 2022-03-29T17:52:07.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.316345215s) for pg: 3.0 2022-03-29T17:52:07.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.316345215s 2022-03-29T17:52:07.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.316345215s (sent 150.316345215s) 2022-03-29T17:52:07.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.316345215s))): submitting ctx 2022-03-29T17:52:07.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,644 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,645 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.332717896s ub 150.336608887s int 16.000000000s))): start 2022-03-29T17:52:07.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,645 [shard 0] osd - peering_event(id=239, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.294967651s ub 150.295852661s int 16.000000000s))): complete 2022-03-29T17:52:07.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,645 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.332717896s ub 150.336608887s int 16.000000000s))): got map 18 2022-03-29T17:52:07.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,645 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.332717896s ub 150.336608887s int 16.000000000s))): pg present 2022-03-29T17:52:07.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.332717896s ub 150.336608887s int 16.000000000s) for pg: 3.1a 2022-03-29T17:52:07.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active proc_lease pg_lease(ru 142.332717896s ub 150.336608887s int 16.000000000s) 2022-03-29T17:52:07.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active no peer_clock_delta_lb -> ruub 150.279479980s 2022-03-29T17:52:07.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active proc_lease readable_until_ub now 150.279479980s 2022-03-29T17:52:07.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.332717896s ub 150.336608887s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - peering_event(id=240, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.317977905s ub 150.316345215s int 16.000000000s))): complete 2022-03-29T17:52:07.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,646 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.316345215s))): complete 2022-03-29T17:52:07.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,649 [shard 0] osd - peering_event(id=241, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.332717896s ub 150.336608887s int 16.000000000s))): complete 2022-03-29T17:52:07.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - peering_event(id=503, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - peering_event(id=504, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - peering_event(id=504, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - peering_event(id=503, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.2 2022-03-29T17:52:07.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.353500366s 2022-03-29T17:52:07.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.353500366s (sent 150.346603394s) 2022-03-29T17:52:07.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - peering_event(id=504, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,650 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.13 2022-03-29T17:52:07.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.347534180s 2022-03-29T17:52:07.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.347534180s (sent 150.347137451s) 2022-03-29T17:52:07.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - peering_event(id=503, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - peering_event(id=504, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - handle_peering_op on 3.1a from 2 2022-03-29T17:52:07.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,651 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.336608887s))): start 2022-03-29T17:52:07.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:52:07.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.353500366s ub 150.346603394s int 16.000000000s))): start 2022-03-29T17:52:07.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.336608887s))): got map 18 2022-03-29T17:52:07.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.336608887s))): pg present 2022-03-29T17:52:07.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.336608887s) for pg: 3.1a 2022-03-29T17:52:07.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until peer osd.2 ruub 150.336608887s 2022-03-29T17:52:07.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until readable_until[_ub] 150.336608887s (sent 150.336608887s) 2022-03-29T17:52:07.681 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,652 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.353500366s ub 150.346603394s int 16.000000000s))): got map 18 2022-03-29T17:52:07.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.353500366s ub 150.346603394s int 16.000000000s))): pg present 2022-03-29T17:52:07.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.353500366s ub 150.346603394s int 16.000000000s) for pg: 3.2 2022-03-29T17:52:07.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.353500366s ub 150.346603394s int 16.000000000s) 2022-03-29T17:52:07.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.286254883s 2022-03-29T17:52:07.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.286254883s 2022-03-29T17:52:07.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.353500366s ub 150.346603394s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.336608887s))): submitting ctx 2022-03-29T17:52:07.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,653 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,654 [shard 0] osd - peering_event(id=503, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,654 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.336608887s))): complete 2022-03-29T17:52:07.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,654 [shard 0] osd - peering_event(id=242, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.353500366s ub 150.346603394s int 16.000000000s))): complete 2022-03-29T17:52:07.687 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,657 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:07.688 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,657 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:07.688 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,658 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:07.688 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:07,658 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 0.4 seconds ... 2022-03-29T17:52:07.688 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,658 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:07.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,680 [shard 0] osd - peering_event(id=506, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,680 [shard 0] osd - peering_event(id=507, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,680 [shard 0] osd - peering_event(id=506, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,680 [shard 0] osd - peering_event(id=507, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,680 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.d 2022-03-29T17:52:07.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean recalc_readable_until peer osd.0 ruub 142.373336792s 2022-03-29T17:52:07.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean recalc_readable_until readable_until[_ub] 142.373336792s (sent 150.376907349s) 2022-03-29T17:52:07.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - peering_event(id=507, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.4 2022-03-29T17:52:07.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 142.370239258s 2022-03-29T17:52:07.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.370239258s (sent 150.377182007s) 2022-03-29T17:52:07.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - peering_event(id=506, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - peering_event(id=507, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,681 [shard 0] osd - handle_peering_op on 3.2 from 2 2022-03-29T17:52:07.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:52:07.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.370239258s ub 150.377182007s int 16.000000000s))): start 2022-03-29T17:52:07.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.370239258s ub 150.377182007s int 16.000000000s))): got map 18 2022-03-29T17:52:07.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.370239258s ub 150.377182007s int 16.000000000s))): pg present 2022-03-29T17:52:07.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.370239258s ub 150.377182007s int 16.000000000s) for pg: 3.4 2022-03-29T17:52:07.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 142.370239258s ub 150.377182007s int 16.000000000s) 2022-03-29T17:52:07.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 150.315765381s 2022-03-29T17:52:07.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 150.315765381s 2022-03-29T17:52:07.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.370239258s ub 150.377182007s int 16.000000000s))): submitting ctx 2022-03-29T17:52:07.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,682 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,683 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.346603394s))): start 2022-03-29T17:52:07.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:07,683 [shard 0] osd - peering_event(id=243, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 142.370239258s ub 150.377182007s int 16.000000000s))): complete 2022-03-29T17:52:07.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,683 [shard 0] osd - peering_event(id=506, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,684 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.346603394s))): got map 18 2022-03-29T17:52:07.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,684 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.346603394s))): pg present 2022-03-29T17:52:07.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,684 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.346603394s) for pg: 3.2 2022-03-29T17:52:07.703 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,684 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.346603394s 2022-03-29T17:52:07.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,684 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.346603394s (sent 150.346603394s) 2022-03-29T17:52:07.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,684 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,684 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.346603394s))): submitting ctx 2022-03-29T17:52:07.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,684 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.704 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,685 [shard 0] osd - handle_peering_op on 3.4 from 2 2022-03-29T17:52:07.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,685 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.377182007s))): start 2022-03-29T17:52:07.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,685 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.377182007s))): got map 18 2022-03-29T17:52:07.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,685 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.377182007s))): pg present 2022-03-29T17:52:07.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,685 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.346603394s))): complete 2022-03-29T17:52:07.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,686 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.377182007s) for pg: 3.4 2022-03-29T17:52:07.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,686 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.377182007s 2022-03-29T17:52:07.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,686 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.377182007s (sent 150.377182007s) 2022-03-29T17:52:07.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,686 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,686 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.377182007s))): submitting ctx 2022-03-29T17:52:07.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,686 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,686 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 150.377182007s))): complete 2022-03-29T17:52:07.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,700 [shard 0] osd - peering_event(id=510, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:07.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,700 [shard 0] osd - peering_event(id=510, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:07.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,700 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.10 2022-03-29T17:52:07.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,700 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.389938354s 2022-03-29T17:52:07.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,700 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.389938354s (sent 150.396652222s) 2022-03-29T17:52:07.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,700 [shard 0] osd - log is not dirty 2022-03-29T17:52:07.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,700 [shard 0] osd - peering_event(id=510, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:07.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,700 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:07.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:07,701 [shard 0] osd - peering_event(id=510, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:07.864 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:07,862 [shard 0] alienstore - stat 2022-03-29T17:52:07.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:07,869 [shard 0] alienstore - stat 2022-03-29T17:52:08.061 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,058 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:08.061 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,058 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:08.062 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,058 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:08.062 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:08,058 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 0.8 seconds ... 2022-03-29T17:52:08.063 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,059 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:08.247 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:08,245 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:08.247 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:08,245 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:08.248 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:08,245 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:08.248 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:08,245 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] waiting 1.6 seconds ... 2022-03-29T17:52:08.248 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:08,246 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:08.861 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,859 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:08.861 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,859 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:08.861 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,859 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:08.861 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:08,859 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 1.6 seconds ... 2022-03-29T17:52:08.862 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,859 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:08.870 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:08,869 [shard 0] alienstore - stat 2022-03-29T17:52:08.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:08,876 [shard 0] alienstore - stat 2022-03-29T17:52:09.230 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2022-03-29T17:52:09.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,299 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:09.302 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,300 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:09.303 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:09,300 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 6.4 seconds ... 2022-03-29T17:52:09.303 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,300 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:09.303 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,300 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:09.303 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:09,300 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 6.4 seconds ... 2022-03-29T17:52:09.305 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:09,301 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:09.305 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:09,301 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:09.305 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:09,301 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:09.305 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:09,301 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 6.4 seconds ... 2022-03-29T17:52:09.306 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:09,301 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:09.306 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:09,302 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 6.4 seconds ... 2022-03-29T17:52:09.848 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,846 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:09.848 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,846 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:09.848 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,846 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:09.848 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:09,846 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] waiting 3.2 seconds ... 2022-03-29T17:52:09.849 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,846 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:09.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,856 [shard 0] osd - peering_event(id=511, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:09.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,856 [shard 0] osd - peering_event(id=511, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:09.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,856 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:52:09.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,856 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 144.548706055s 2022-03-29T17:52:09.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,856 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.548706055s (sent 152.552536011s) 2022-03-29T17:52:09.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,856 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,856 [shard 0] osd - peering_event(id=511, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:09.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,856 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,857 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:52:09.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,857 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 144.548706055s ub 152.552536011s int 16.000000000s))): start 2022-03-29T17:52:09.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,857 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 144.548706055s ub 152.552536011s int 16.000000000s))): got map 18 2022-03-29T17:52:09.862 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,857 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 144.548706055s ub 152.552536011s int 16.000000000s))): pg present 2022-03-29T17:52:09.863 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,857 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 144.548706055s ub 152.552536011s int 16.000000000s) for pg: 2.3 2022-03-29T17:52:09.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,858 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 144.548706055s ub 152.552536011s int 16.000000000s) 2022-03-29T17:52:09.864 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,858 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 152.491119385s 2022-03-29T17:52:09.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,858 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 152.491119385s 2022-03-29T17:52:09.865 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,858 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,858 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 144.548706055s ub 152.552536011s int 16.000000000s))): submitting ctx 2022-03-29T17:52:09.866 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,858 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,859 [shard 0] osd - peering_event(id=511, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:09.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,859 [shard 0] osd - peering_event(id=244, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 144.548706055s ub 152.552536011s int 16.000000000s))): complete 2022-03-29T17:52:09.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,863 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:09.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,863 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:09.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,863 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:52:09.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,863 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 144.489608765s 2022-03-29T17:52:09.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,863 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.489608765s (sent 152.496231079s) 2022-03-29T17:52:09.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,863 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,863 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:09.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,863 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,864 [shard 0] osd - peering_event(id=245, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:09.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,873 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:09.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,873 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:09.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,873 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:52:09.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,873 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 144.499664307s 2022-03-29T17:52:09.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,873 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.499664307s (sent 152.506469727s) 2022-03-29T17:52:09.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,873 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,873 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:09.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,873 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,874 [shard 0] osd - peering_event(id=246, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:09.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,876 [shard 0] osd - peering_event(id=512, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:09.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,876 [shard 0] osd - peering_event(id=513, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:09.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,876 [shard 0] osd - peering_event(id=514, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:09.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,876 [shard 0] osd - peering_event(id=515, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:09.883 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:09,876 [shard 0] alienstore - stat 2022-03-29T17:52:09.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - peering_event(id=512, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:09.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - peering_event(id=513, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:09.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - peering_event(id=514, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:09.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - peering_event(id=515, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:09.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:52:09.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.568969727s 2022-03-29T17:52:09.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.568969727s (sent 152.573379517s) 2022-03-29T17:52:09.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - peering_event(id=512, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:09.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:52:09.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569427490s 2022-03-29T17:52:09.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569427490s (sent 152.573638916s) 2022-03-29T17:52:09.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - peering_event(id=514, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:09.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,877 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:52:09.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569213867s 2022-03-29T17:52:09.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569213867s (sent 152.573883057s) 2022-03-29T17:52:09.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - peering_event(id=513, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:09.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:52:09.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569808960s 2022-03-29T17:52:09.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569808960s (sent 152.574203491s) 2022-03-29T17:52:09.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - peering_event(id=515, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:09.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - peering_event(id=512, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:09.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,878 [shard 0] osd - peering_event(id=514, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:09.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,881 [shard 0] osd - peering_event(id=513, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:09.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,881 [shard 0] osd - peering_event(id=515, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:09.891 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:09,883 [shard 0] alienstore - stat 2022-03-29T17:52:09.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,898 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:52:09.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,898 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 152.552536011s))): start 2022-03-29T17:52:09.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 152.552536011s))): got map 18 2022-03-29T17:52:09.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 152.552536011s))): pg present 2022-03-29T17:52:09.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:52:09.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.489608765s ub 152.496231079s int 16.000000000s))): start 2022-03-29T17:52:09.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 152.552536011s) for pg: 2.3 2022-03-29T17:52:09.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 152.552536011s 2022-03-29T17:52:09.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 152.552536011s (sent 152.552536011s) 2022-03-29T17:52:09.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 152.552536011s))): submitting ctx 2022-03-29T17:52:09.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,899 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.489608765s ub 152.496231079s int 16.000000000s))): got map 18 2022-03-29T17:52:09.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.489608765s ub 152.496231079s int 16.000000000s))): pg present 2022-03-29T17:52:09.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:52:09.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.499664307s ub 152.506469727s int 16.000000000s))): start 2022-03-29T17:52:09.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.489608765s ub 152.496231079s int 16.000000000s) for pg: 2.0 2022-03-29T17:52:09.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 144.489608765s ub 152.496231079s int 16.000000000s) 2022-03-29T17:52:09.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 152.596649170s 2022-03-29T17:52:09.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 152.596649170s 2022-03-29T17:52:09.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.489608765s ub 152.496231079s int 16.000000000s))): submitting ctx 2022-03-29T17:52:09.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,900 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.499664307s ub 152.506469727s int 16.000000000s))): got map 18 2022-03-29T17:52:09.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 152.552536011s))): complete 2022-03-29T17:52:09.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.499664307s ub 152.506469727s int 16.000000000s))): pg present 2022-03-29T17:52:09.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.489608765s ub 152.496231079s int 16.000000000s))): complete 2022-03-29T17:52:09.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.499664307s ub 152.506469727s int 16.000000000s) for pg: 2.1 2022-03-29T17:52:09.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 144.499664307s ub 152.506469727s int 16.000000000s) 2022-03-29T17:52:09.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 152.597564697s 2022-03-29T17:52:09.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 152.597564697s 2022-03-29T17:52:09.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.499664307s ub 152.506469727s int 16.000000000s))): submitting ctx 2022-03-29T17:52:09.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,901 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:09,904 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 144.499664307s ub 152.506469727s int 16.000000000s))): complete 2022-03-29T17:52:09.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,939 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:52:09.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.496231079s))): start 2022-03-29T17:52:09.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.496231079s))): got map 18 2022-03-29T17:52:09.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.496231079s))): pg present 2022-03-29T17:52:09.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:52:09.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.506469727s))): start 2022-03-29T17:52:09.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.496231079s) for pg: 2.0 2022-03-29T17:52:09.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 152.496231079s 2022-03-29T17:52:09.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 152.496231079s (sent 152.496231079s) 2022-03-29T17:52:09.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.496231079s))): submitting ctx 2022-03-29T17:52:09.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,940 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.506469727s))): got map 18 2022-03-29T17:52:09.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.506469727s))): pg present 2022-03-29T17:52:09.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.506469727s) for pg: 2.1 2022-03-29T17:52:09.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 152.506469727s 2022-03-29T17:52:09.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 152.506469727s (sent 152.506469727s) 2022-03-29T17:52:09.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - log is not dirty 2022-03-29T17:52:09.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.506469727s))): submitting ctx 2022-03-29T17:52:09.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:09.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,941 [shard 0] osd - peering_event(id=247, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.496231079s))): complete 2022-03-29T17:52:09.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:09,944 [shard 0] osd - peering_event(id=248, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 152.506469727s))): complete 2022-03-29T17:52:10.463 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:10,459 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:10.463 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:10,459 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:10.464 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:10,460 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:10.464 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:10,460 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 3.2 seconds ... 2022-03-29T17:52:10.465 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:10,460 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:10.885 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:10,884 [shard 0] alienstore - stat 2022-03-29T17:52:10.891 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:10,889 [shard 0] alienstore - stat 2022-03-29T17:52:11.893 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:11,891 [shard 0] alienstore - stat 2022-03-29T17:52:11.898 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:11,896 [shard 0] alienstore - stat 2022-03-29T17:52:12.306 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:12,303 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is still pending... 2022-03-29T17:52:12.306 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,304 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:12.306 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,304 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=15684760443689780352 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:12.307 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,304 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:12.307 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,304 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=17857566129473309735 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:12.307 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,304 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:12.308 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,306 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:12.309 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,306 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.309 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:12,307 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:12.309 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,307 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.309 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:12,307 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:12.351 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:12,349 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is still pending... 2022-03-29T17:52:12.352 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,349 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:12.352 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,349 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=18075862943584853051 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:12.352 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,349 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:12.352 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,349 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=5783385256795883373 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:12.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:12.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,350 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:12.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,350 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.355 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:12,350 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:12.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.356 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:12,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:12.509 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,507 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:12.510 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,507 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:12.510 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,507 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.510 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:12,507 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:12.510 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,507 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.511 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:12,507 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:12.553 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,550 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:12.553 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,551 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:12.553 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,551 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.554 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:12,551 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:12.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,551 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.555 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:12,551 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:12.899 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,898 [shard 0] alienstore - stat 2022-03-29T17:52:12.904 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,903 [shard 0] alienstore - stat 2022-03-29T17:52:12.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,907 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:12.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,908 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:12.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,908 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.910 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:12,908 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:12.911 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:12,908 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.911 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:12,908 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:12.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,951 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:12.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,952 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:12.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,952 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.955 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:12,952 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:12.955 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:12,952 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:12.955 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:12,952 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:13.048 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,046 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:13.049 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,046 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:13.049 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,047 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:13.049 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:13,047 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] waiting 6.4 seconds ... 2022-03-29T17:52:13.049 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,047 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:13.662 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,660 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:13.662 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,660 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:13.663 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,661 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:13.663 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:13,661 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 6.4 seconds ... 2022-03-29T17:52:13.663 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,661 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:13.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,708 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:13.710 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,709 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:13.711 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,709 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:13.711 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:13,709 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:13.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,709 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:13.712 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:13,709 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:13.754 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,752 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:13.754 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:13.755 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,753 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:13.755 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:13,753 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:13.756 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:13.756 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:13,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:13.907 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:13,905 [shard 0] alienstore - stat 2022-03-29T17:52:13.911 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:13,910 [shard 0] alienstore - stat 2022-03-29T17:52:14.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2022-03-29T17:52:14.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,851 [shard 0] osd - peering_event(id=519, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:52:14.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,851 [shard 0] osd - peering_event(id=519, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:52:14.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,851 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:52:14.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,851 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 141.530502319s 2022-03-29T17:52:14.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,851 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 141.530502319s (sent 157.547515869s) 2022-03-29T17:52:14.854 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,851 [shard 0] osd - log is not dirty 2022-03-29T17:52:14.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,851 [shard 0] osd - peering_event(id=519, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:52:14.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,851 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:14.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:14,852 [shard 0] osd - peering_event(id=519, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:52:14.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:14,912 [shard 0] alienstore - stat 2022-03-29T17:52:14.918 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:14,917 [shard 0] alienstore - stat 2022-03-29T17:52:15.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:15,309 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:15.312 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:15,309 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:15.312 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:15,310 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:15.312 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:15,310 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 3.2 seconds ... 2022-03-29T17:52:15.313 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:15,310 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:15.313 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:15,310 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 3.2 seconds ... 2022-03-29T17:52:15.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:15,353 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:15.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:15,353 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:15.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:15,354 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:15.358 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:15,354 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 3.2 seconds ... 2022-03-29T17:52:15.358 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:15,354 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:15.358 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:15,354 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 3.2 seconds ... 2022-03-29T17:52:15.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,460 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,460 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,461 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1d 2022-03-29T17:52:15.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,461 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.095642090s 2022-03-29T17:52:15.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,461 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.095642090s (sent 158.094192505s) 2022-03-29T17:52:15.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,461 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,461 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,461 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,462 [shard 0] osd - peering_event(id=249, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,480 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,480 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,481 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:52:15.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,481 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.110015869s 2022-03-29T17:52:15.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,481 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.110015869s (sent 158.114166260s) 2022-03-29T17:52:15.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,481 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.484 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,481 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.485 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,481 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,482 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:52:15.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,482 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.110015869s ub 158.114166260s int 16.000000000s))): start 2022-03-29T17:52:15.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,482 [shard 0] osd - peering_event(id=250, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,482 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.110015869s ub 158.114166260s int 16.000000000s))): got map 18 2022-03-29T17:52:15.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,483 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.110015869s ub 158.114166260s int 16.000000000s))): pg present 2022-03-29T17:52:15.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,483 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.110015869s ub 158.114166260s int 16.000000000s) for pg: 3.8 2022-03-29T17:52:15.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,483 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.110015869s ub 158.114166260s int 16.000000000s) 2022-03-29T17:52:15.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,483 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.179504395s 2022-03-29T17:52:15.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,483 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.179504395s 2022-03-29T17:52:15.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,483 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,483 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.110015869s ub 158.114166260s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,483 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,484 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.110015869s ub 158.114166260s int 16.000000000s))): complete 2022-03-29T17:52:15.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,490 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,490 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,490 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:52:15.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 150.120346069s 2022-03-29T17:52:15.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 150.120346069s (sent 158.124221802s) 2022-03-29T17:52:15.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.496 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.18 2022-03-29T17:52:15.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.121215820s 2022-03-29T17:52:15.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.121215820s (sent 158.124786377s) 2022-03-29T17:52:15.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,491 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - handle_peering_op on 3.8 from 1 2022-03-29T17:52:15.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.114166260s))): start 2022-03-29T17:52:15.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.114166260s))): got map 18 2022-03-29T17:52:15.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - peering_event(id=252, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.114166260s))): pg present 2022-03-29T17:52:15.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.114166260s) for pg: 3.8 2022-03-29T17:52:15.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.114166260s 2022-03-29T17:52:15.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.114166260s (sent 158.114166260s) 2022-03-29T17:52:15.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.114166260s))): submitting ctx 2022-03-29T17:52:15.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,492 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,493 [shard 0] osd - peering_event(id=251, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,493 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:52:15.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,493 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.120346069s ub 158.124221802s int 16.000000000s))): start 2022-03-29T17:52:15.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,493 [shard 0] osd - peering_event(id=253, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.114166260s))): complete 2022-03-29T17:52:15.503 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.120346069s ub 158.124221802s int 16.000000000s))): got map 18 2022-03-29T17:52:15.503 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.120346069s ub 158.124221802s int 16.000000000s))): pg present 2022-03-29T17:52:15.503 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.120346069s ub 158.124221802s int 16.000000000s) for pg: 3.7 2022-03-29T17:52:15.504 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active proc_lease pg_lease(ru 150.120346069s ub 158.124221802s int 16.000000000s) 2022-03-29T17:52:15.504 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active no peer_clock_delta_lb -> ruub 158.190567017s 2022-03-29T17:52:15.504 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active proc_lease readable_until_ub now 158.190567017s 2022-03-29T17:52:15.504 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.505 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.120346069s ub 158.124221802s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.505 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,494 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.505 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,495 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.120346069s ub 158.124221802s int 16.000000000s))): complete 2022-03-29T17:52:15.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:52:15.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.149948120s 2022-03-29T17:52:15.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.149948120s (sent 158.154312134s) 2022-03-29T17:52:15.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - handle_peering_op on 3.7 from 1 2022-03-29T17:52:15.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,521 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.124221802s))): start 2022-03-29T17:52:15.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.124221802s))): got map 18 2022-03-29T17:52:15.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - peering_event(id=254, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.124221802s))): pg present 2022-03-29T17:52:15.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.124221802s) for pg: 3.7 2022-03-29T17:52:15.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 158.124221802s 2022-03-29T17:52:15.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 158.124221802s (sent 158.124221802s) 2022-03-29T17:52:15.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.124221802s))): submitting ctx 2022-03-29T17:52:15.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,522 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,523 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:52:15.535 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,523 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.149948120s ub 158.154312134s int 16.000000000s))): start 2022-03-29T17:52:15.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,523 [shard 0] osd - peering_event(id=255, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.124221802s))): complete 2022-03-29T17:52:15.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.149948120s ub 158.154312134s int 16.000000000s))): got map 18 2022-03-29T17:52:15.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.149948120s ub 158.154312134s int 16.000000000s))): pg present 2022-03-29T17:52:15.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.149948120s ub 158.154312134s int 16.000000000s) for pg: 3.1e 2022-03-29T17:52:15.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.149948120s ub 158.154312134s int 16.000000000s) 2022-03-29T17:52:15.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.220306396s 2022-03-29T17:52:15.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.220306396s 2022-03-29T17:52:15.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.149948120s ub 158.154312134s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,524 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,525 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.149948120s ub 158.154312134s int 16.000000000s))): complete 2022-03-29T17:52:15.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,541 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,541 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,541 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.e 2022-03-29T17:52:15.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,541 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.0 ruub 142.164947510s 2022-03-29T17:52:15.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,541 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 142.164947510s (sent 158.174484253s) 2022-03-29T17:52:15.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,541 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,541 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,541 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,542 [shard 0] osd - peering_event(id=256, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,551 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,551 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,551 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:52:15.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,551 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.180313110s 2022-03-29T17:52:15.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,551 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.180313110s (sent 158.184570312s) 2022-03-29T17:52:15.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,551 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,551 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,551 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - handle_peering_op on 3.1e from 1 2022-03-29T17:52:15.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.154312134s))): start 2022-03-29T17:52:15.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - peering_event(id=257, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.154312134s))): got map 18 2022-03-29T17:52:15.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.154312134s))): pg present 2022-03-29T17:52:15.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.154312134s) for pg: 3.1e 2022-03-29T17:52:15.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.154312134s 2022-03-29T17:52:15.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.154312134s (sent 158.154312134s) 2022-03-29T17:52:15.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.154312134s))): submitting ctx 2022-03-29T17:52:15.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,552 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,553 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:52:15.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,553 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.180313110s ub 158.184570312s int 16.000000000s))): start 2022-03-29T17:52:15.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,553 [shard 0] osd - peering_event(id=258, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.154312134s))): complete 2022-03-29T17:52:15.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.180313110s ub 158.184570312s int 16.000000000s))): got map 18 2022-03-29T17:52:15.572 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.180313110s ub 158.184570312s int 16.000000000s))): pg present 2022-03-29T17:52:15.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.180313110s ub 158.184570312s int 16.000000000s) for pg: 3.16 2022-03-29T17:52:15.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.180313110s ub 158.184570312s int 16.000000000s) 2022-03-29T17:52:15.573 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.250503540s 2022-03-29T17:52:15.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.250503540s 2022-03-29T17:52:15.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.180313110s ub 158.184570312s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,554 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,555 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.180313110s ub 158.184570312s int 16.000000000s))): complete 2022-03-29T17:52:15.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,556 [shard 0] osd - peering_event(id=524, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,556 [shard 0] osd - peering_event(id=524, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,556 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.19 2022-03-29T17:52:15.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,556 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.245559692s 2022-03-29T17:52:15.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,556 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.245559692s (sent 158.252090454s) 2022-03-29T17:52:15.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,556 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,556 [shard 0] osd - peering_event(id=524, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,556 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,557 [shard 0] osd - peering_event(id=524, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,561 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,561 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,561 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:52:15.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,561 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.190109253s 2022-03-29T17:52:15.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,561 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.190109253s (sent 158.194564819s) 2022-03-29T17:52:15.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,561 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,561 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,561 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,562 [shard 0] osd - handle_peering_op on 3.16 from 1 2022-03-29T17:52:15.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,562 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.184570312s))): start 2022-03-29T17:52:15.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,562 [shard 0] osd - peering_event(id=259, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,562 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.184570312s))): got map 18 2022-03-29T17:52:15.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,562 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.184570312s))): pg present 2022-03-29T17:52:15.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,562 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:52:15.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,562 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.245559692s ub 158.252090454s int 16.000000000s))): start 2022-03-29T17:52:15.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.184570312s) for pg: 3.16 2022-03-29T17:52:15.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.184570312s 2022-03-29T17:52:15.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.184570312s (sent 158.184570312s) 2022-03-29T17:52:15.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.184570312s))): submitting ctx 2022-03-29T17:52:15.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.245559692s ub 158.252090454s int 16.000000000s))): got map 18 2022-03-29T17:52:15.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.245559692s ub 158.252090454s int 16.000000000s))): pg present 2022-03-29T17:52:15.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.245559692s ub 158.252090454s int 16.000000000s) for pg: 3.19 2022-03-29T17:52:15.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.245559692s ub 158.252090454s int 16.000000000s) 2022-03-29T17:52:15.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.196533203s 2022-03-29T17:52:15.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.196533203s 2022-03-29T17:52:15.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.245559692s ub 158.252090454s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,563 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,564 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:52:15.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,564 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.190109253s ub 158.194564819s int 16.000000000s))): start 2022-03-29T17:52:15.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,564 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.190109253s ub 158.194564819s int 16.000000000s))): got map 18 2022-03-29T17:52:15.592 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,564 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.190109253s ub 158.194564819s int 16.000000000s))): pg present 2022-03-29T17:52:15.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,564 [shard 0] osd - peering_event(id=260, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.184570312s))): complete 2022-03-29T17:52:15.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,564 [shard 0] osd - peering_event(id=261, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.245559692s ub 158.252090454s int 16.000000000s))): complete 2022-03-29T17:52:15.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,565 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.190109253s ub 158.194564819s int 16.000000000s) for pg: 3.5 2022-03-29T17:52:15.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,565 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.190109253s ub 158.194564819s int 16.000000000s) 2022-03-29T17:52:15.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,565 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.261367798s 2022-03-29T17:52:15.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,565 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.261367798s 2022-03-29T17:52:15.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,565 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,565 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.190109253s ub 158.194564819s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,565 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - handle_peering_op on 3.5 from 1 2022-03-29T17:52:15.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.194564819s))): start 2022-03-29T17:52:15.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.194564819s))): got map 18 2022-03-29T17:52:15.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.194564819s))): pg present 2022-03-29T17:52:15.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.194564819s) for pg: 3.5 2022-03-29T17:52:15.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.194564819s 2022-03-29T17:52:15.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.194564819s (sent 158.194564819s) 2022-03-29T17:52:15.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.194564819s))): submitting ctx 2022-03-29T17:52:15.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,566 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,567 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.190109253s ub 158.194564819s int 16.000000000s))): complete 2022-03-29T17:52:15.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,567 [shard 0] osd - handle_peering_op on 3.19 from 2 2022-03-29T17:52:15.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,567 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.252090454s))): start 2022-03-29T17:52:15.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,567 [shard 0] osd - peering_event(id=262, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.194564819s))): complete 2022-03-29T17:52:15.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,567 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.252090454s))): got map 18 2022-03-29T17:52:15.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,567 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.252090454s))): pg present 2022-03-29T17:52:15.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,568 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.252090454s) for pg: 3.19 2022-03-29T17:52:15.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,568 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.252090454s 2022-03-29T17:52:15.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,568 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.252090454s (sent 158.252090454s) 2022-03-29T17:52:15.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,568 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,568 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.252090454s))): submitting ctx 2022-03-29T17:52:15.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,568 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,568 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.252090454s))): complete 2022-03-29T17:52:15.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,591 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,591 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,591 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:52:15.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,591 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 150.220336914s 2022-03-29T17:52:15.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,591 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.220336914s (sent 158.224838257s) 2022-03-29T17:52:15.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,591 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,591 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,592 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,592 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:52:15.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,592 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.220336914s ub 158.224838257s int 16.000000000s))): start 2022-03-29T17:52:15.609 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.220336914s ub 158.224838257s int 16.000000000s))): got map 18 2022-03-29T17:52:15.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.220336914s ub 158.224838257s int 16.000000000s))): pg present 2022-03-29T17:52:15.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.220336914s ub 158.224838257s int 16.000000000s) for pg: 3.11 2022-03-29T17:52:15.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.220336914s ub 158.224838257s int 16.000000000s) 2022-03-29T17:52:15.610 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.289215088s 2022-03-29T17:52:15.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.289215088s 2022-03-29T17:52:15.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.220336914s ub 158.224838257s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.611 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,593 [shard 0] osd - peering_event(id=263, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,594 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 150.220336914s ub 158.224838257s int 16.000000000s))): complete 2022-03-29T17:52:15.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,607 [shard 0] osd - peering_event(id=528, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,607 [shard 0] osd - peering_event(id=528, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,607 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.14 2022-03-29T17:52:15.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,607 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.295852661s 2022-03-29T17:52:15.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,607 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.295852661s (sent 158.303329468s) 2022-03-29T17:52:15.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,607 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,607 [shard 0] osd - peering_event(id=528, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,607 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,608 [shard 0] osd - peering_event(id=528, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - peering_event(id=529, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - peering_event(id=530, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - peering_event(id=530, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - peering_event(id=529, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1c 2022-03-29T17:52:15.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean recalc_readable_until peer osd.0 ruub 142.314071655s 2022-03-29T17:52:15.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean recalc_readable_until readable_until[_ub] 142.314071655s (sent 158.323715210s) 2022-03-29T17:52:15.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - peering_event(id=529, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,627 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.0 2022-03-29T17:52:15.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,628 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.316345215s 2022-03-29T17:52:15.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,628 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.316345215s (sent 158.323913574s) 2022-03-29T17:52:15.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,628 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.638 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,628 [shard 0] osd - peering_event(id=530, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,628 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,628 [shard 0] osd - peering_event(id=530, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.639 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,629 [shard 0] osd - peering_event(id=529, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,633 [shard 0] osd - handle_peering_op on 3.11 from 1 2022-03-29T17:52:15.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,633 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.224838257s))): start 2022-03-29T17:52:15.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,633 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.224838257s))): got map 18 2022-03-29T17:52:15.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,633 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.224838257s))): pg present 2022-03-29T17:52:15.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:52:15.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.295852661s ub 158.303329468s int 16.000000000s))): start 2022-03-29T17:52:15.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.224838257s) for pg: 3.11 2022-03-29T17:52:15.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.224838257s 2022-03-29T17:52:15.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.224838257s (sent 158.224838257s) 2022-03-29T17:52:15.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.224838257s))): submitting ctx 2022-03-29T17:52:15.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.295852661s ub 158.303329468s int 16.000000000s))): got map 18 2022-03-29T17:52:15.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.295852661s ub 158.303329468s int 16.000000000s))): pg present 2022-03-29T17:52:15.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:52:15.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,634 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.316345215s ub 158.323913574s int 16.000000000s))): start 2022-03-29T17:52:15.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - peering_event(id=264, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 158.224838257s))): complete 2022-03-29T17:52:15.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.316345215s ub 158.323913574s int 16.000000000s))): got map 18 2022-03-29T17:52:15.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.295852661s ub 158.303329468s int 16.000000000s) for pg: 3.14 2022-03-29T17:52:15.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.295852661s ub 158.303329468s int 16.000000000s) 2022-03-29T17:52:15.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.268402100s 2022-03-29T17:52:15.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.268402100s 2022-03-29T17:52:15.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.295852661s ub 158.303329468s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.316345215s ub 158.323913574s int 16.000000000s))): pg present 2022-03-29T17:52:15.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.316345215s ub 158.323913574s int 16.000000000s) for pg: 3.0 2022-03-29T17:52:15.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.316345215s ub 158.323913574s int 16.000000000s) 2022-03-29T17:52:15.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.268951416s 2022-03-29T17:52:15.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,635 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.268951416s 2022-03-29T17:52:15.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,636 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,636 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.316345215s ub 158.323913574s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,636 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,636 [shard 0] osd - peering_event(id=265, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.295852661s ub 158.303329468s int 16.000000000s))): complete 2022-03-29T17:52:15.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,636 [shard 0] osd - peering_event(id=266, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.316345215s ub 158.323913574s int 16.000000000s))): complete 2022-03-29T17:52:15.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - peering_event(id=531, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - peering_event(id=532, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - peering_event(id=532, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - peering_event(id=531, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1a 2022-03-29T17:52:15.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until peer osd.2 ruub 150.336608887s 2022-03-29T17:52:15.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until readable_until[_ub] 150.336608887s (sent 158.343627930s) 2022-03-29T17:52:15.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - peering_event(id=531, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,647 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.b 2022-03-29T17:52:15.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.336990356s 2022-03-29T17:52:15.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.336990356s (sent 158.343963623s) 2022-03-29T17:52:15.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - peering_event(id=532, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:52:15.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.336608887s ub 158.343627930s int 16.000000000s))): start 2022-03-29T17:52:15.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.336608887s ub 158.343627930s int 16.000000000s))): got map 18 2022-03-29T17:52:15.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,648 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.336608887s ub 158.343627930s int 16.000000000s))): pg present 2022-03-29T17:52:15.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.336608887s ub 158.343627930s int 16.000000000s) for pg: 3.1a 2022-03-29T17:52:15.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active proc_lease pg_lease(ru 150.336608887s ub 158.343627930s int 16.000000000s) 2022-03-29T17:52:15.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active no peer_clock_delta_lb -> ruub 158.282196045s 2022-03-29T17:52:15.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active proc_lease readable_until_ub now 158.282196045s 2022-03-29T17:52:15.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.336608887s ub 158.343627930s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - peering_event(id=267, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.336608887s ub 158.343627930s int 16.000000000s))): complete 2022-03-29T17:52:15.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,649 [shard 0] osd - peering_event(id=531, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,650 [shard 0] osd - peering_event(id=532, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,650 [shard 0] osd - handle_peering_op on 3.14 from 2 2022-03-29T17:52:15.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,650 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.303329468s))): start 2022-03-29T17:52:15.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.303329468s))): got map 18 2022-03-29T17:52:15.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.303329468s))): pg present 2022-03-29T17:52:15.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - handle_peering_op on 3.0 from 2 2022-03-29T17:52:15.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.323913574s))): start 2022-03-29T17:52:15.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.303329468s) for pg: 3.14 2022-03-29T17:52:15.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.303329468s 2022-03-29T17:52:15.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.303329468s (sent 158.303329468s) 2022-03-29T17:52:15.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.303329468s))): submitting ctx 2022-03-29T17:52:15.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,651 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,652 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.323913574s))): got map 18 2022-03-29T17:52:15.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,652 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.323913574s))): pg present 2022-03-29T17:52:15.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,652 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.303329468s))): complete 2022-03-29T17:52:15.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.323913574s) for pg: 3.0 2022-03-29T17:52:15.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.323913574s 2022-03-29T17:52:15.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.323913574s (sent 158.323913574s) 2022-03-29T17:52:15.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.323913574s))): submitting ctx 2022-03-29T17:52:15.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - handle_peering_op on 3.1a from 2 2022-03-29T17:52:15.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.343627930s))): start 2022-03-29T17:52:15.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.343627930s))): got map 18 2022-03-29T17:52:15.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,653 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.343627930s))): pg present 2022-03-29T17:52:15.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,654 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.323913574s))): complete 2022-03-29T17:52:15.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,654 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.343627930s) for pg: 3.1a 2022-03-29T17:52:15.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,654 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until peer osd.2 ruub 158.343627930s 2022-03-29T17:52:15.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,654 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until readable_until[_ub] 158.343627930s (sent 158.343627930s) 2022-03-29T17:52:15.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,654 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,654 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.343627930s))): submitting ctx 2022-03-29T17:52:15.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,654 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,654 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.343627930s))): complete 2022-03-29T17:52:15.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,657 [shard 0] osd - peering_event(id=536, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,657 [shard 0] osd - peering_event(id=537, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,657 [shard 0] osd - peering_event(id=536, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,657 [shard 0] osd - peering_event(id=537, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.676 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,657 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.2 2022-03-29T17:52:15.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,657 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.346603394s 2022-03-29T17:52:15.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,657 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.346603394s (sent 158.353729248s) 2022-03-29T17:52:15.677 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - peering_event(id=536, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.678 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.13 2022-03-29T17:52:15.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.347534180s 2022-03-29T17:52:15.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.347534180s (sent 158.354171753s) 2022-03-29T17:52:15.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.679 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - peering_event(id=537, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,658 [shard 0] osd - peering_event(id=536, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.680 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,659 [shard 0] osd - peering_event(id=537, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - peering_event(id=538, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.692 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - peering_event(id=539, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - peering_event(id=539, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - peering_event(id=538, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.d 2022-03-29T17:52:15.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean recalc_readable_until peer osd.0 ruub 142.373336792s 2022-03-29T17:52:15.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean recalc_readable_until readable_until[_ub] 142.373336792s (sent 158.383789062s) 2022-03-29T17:52:15.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,687 [shard 0] osd - peering_event(id=538, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.4 2022-03-29T17:52:15.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 150.377182007s 2022-03-29T17:52:15.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 150.377182007s (sent 158.384033203s) 2022-03-29T17:52:15.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - peering_event(id=539, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - peering_event(id=538, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,688 [shard 0] osd - peering_event(id=539, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,692 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:52:15.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,692 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.346603394s ub 158.353729248s int 16.000000000s))): start 2022-03-29T17:52:15.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,692 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.346603394s ub 158.353729248s int 16.000000000s))): got map 18 2022-03-29T17:52:15.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,692 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.346603394s ub 158.353729248s int 16.000000000s))): pg present 2022-03-29T17:52:15.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.346603394s ub 158.353729248s int 16.000000000s) for pg: 3.2 2022-03-29T17:52:15.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.346603394s ub 158.353729248s int 16.000000000s) 2022-03-29T17:52:15.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.326248169s 2022-03-29T17:52:15.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.326248169s 2022-03-29T17:52:15.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.346603394s ub 158.353729248s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:52:15.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.377182007s ub 158.384033203s int 16.000000000s))): start 2022-03-29T17:52:15.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.377182007s ub 158.384033203s int 16.000000000s))): got map 18 2022-03-29T17:52:15.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,693 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.377182007s ub 158.384033203s int 16.000000000s))): pg present 2022-03-29T17:52:15.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,694 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.377182007s ub 158.384033203s int 16.000000000s) for pg: 3.4 2022-03-29T17:52:15.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,694 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 150.377182007s ub 158.384033203s int 16.000000000s) 2022-03-29T17:52:15.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,694 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 158.327178955s 2022-03-29T17:52:15.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,694 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 158.327178955s 2022-03-29T17:52:15.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,694 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,694 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.377182007s ub 158.384033203s int 16.000000000s))): submitting ctx 2022-03-29T17:52:15.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,694 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,694 [shard 0] osd - peering_event(id=268, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.346603394s ub 158.353729248s int 16.000000000s))): complete 2022-03-29T17:52:15.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:15,695 [shard 0] osd - peering_event(id=269, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 150.377182007s ub 158.384033203s int 16.000000000s))): complete 2022-03-29T17:52:15.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,707 [shard 0] osd - peering_event(id=540, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:15.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,707 [shard 0] osd - peering_event(id=540, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:15.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,707 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.10 2022-03-29T17:52:15.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,707 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.389938354s 2022-03-29T17:52:15.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,707 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.389938354s (sent 158.403808594s) 2022-03-29T17:52:15.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,707 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,708 [shard 0] osd - peering_event(id=540, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:15.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,708 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,708 [shard 0] osd - peering_event(id=540, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:15.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,733 [shard 0] osd - handle_peering_op on 3.2 from 2 2022-03-29T17:52:15.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,733 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.353729248s))): start 2022-03-29T17:52:15.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,734 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.353729248s))): got map 18 2022-03-29T17:52:15.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,734 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.353729248s))): pg present 2022-03-29T17:52:15.741 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,734 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.353729248s) for pg: 3.2 2022-03-29T17:52:15.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,734 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.353729248s 2022-03-29T17:52:15.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,734 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.353729248s (sent 158.353729248s) 2022-03-29T17:52:15.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,734 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,734 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.353729248s))): submitting ctx 2022-03-29T17:52:15.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,734 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - handle_peering_op on 3.4 from 2 2022-03-29T17:52:15.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.384033203s))): start 2022-03-29T17:52:15.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.384033203s))): got map 18 2022-03-29T17:52:15.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.353729248s))): complete 2022-03-29T17:52:15.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.384033203s))): pg present 2022-03-29T17:52:15.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.384033203s) for pg: 3.4 2022-03-29T17:52:15.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.384033203s 2022-03-29T17:52:15.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.384033203s (sent 158.384033203s) 2022-03-29T17:52:15.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - log is not dirty 2022-03-29T17:52:15.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.384033203s))): submitting ctx 2022-03-29T17:52:15.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,735 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:15.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:15,736 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 158.384033203s))): complete 2022-03-29T17:52:15.921 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:15,919 [shard 0] alienstore - stat 2022-03-29T17:52:15.925 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:15,924 [shard 0] alienstore - stat 2022-03-29T17:52:16.931 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:16,926 [shard 0] alienstore - stat 2022-03-29T17:52:16.932 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:16,930 [shard 0] alienstore - stat 2022-03-29T17:52:17.865 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,862 [shard 0] osd - peering_event(id=543, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:17.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,862 [shard 0] osd - peering_event(id=543, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:17.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,863 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:52:17.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,863 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 152.552536011s 2022-03-29T17:52:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,863 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 152.552536011s (sent 160.559005737s) 2022-03-29T17:52:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,863 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,863 [shard 0] osd - peering_event(id=543, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,863 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:52:17.869 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 152.552536011s ub 160.559005737s int 16.000000000s))): start 2022-03-29T17:52:17.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 152.552536011s ub 160.559005737s int 16.000000000s))): got map 18 2022-03-29T17:52:17.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 152.552536011s ub 160.559005737s int 16.000000000s))): pg present 2022-03-29T17:52:17.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 152.552536011s ub 160.559005737s int 16.000000000s) for pg: 2.3 2022-03-29T17:52:17.870 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 152.552536011s ub 160.559005737s int 16.000000000s) 2022-03-29T17:52:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 160.497695923s 2022-03-29T17:52:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 160.497695923s 2022-03-29T17:52:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 152.552536011s ub 160.559005737s int 16.000000000s))): submitting ctx 2022-03-29T17:52:17.872 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,864 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,865 [shard 0] osd - peering_event(id=543, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:17.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,865 [shard 0] osd - peering_event(id=270, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 152.552536011s ub 160.559005737s int 16.000000000s))): complete 2022-03-29T17:52:17.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,867 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:17.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,867 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:17.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,867 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:52:17.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,867 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 152.496231079s 2022-03-29T17:52:17.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,867 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 152.496231079s (sent 160.500320435s) 2022-03-29T17:52:17.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,867 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,867 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:17.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,867 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,868 [shard 0] osd - peering_event(id=271, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:17.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,877 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:17.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,877 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:17.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,877 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:52:17.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,877 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 152.506469727s 2022-03-29T17:52:17.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,877 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 152.506469727s (sent 160.510345459s) 2022-03-29T17:52:17.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,877 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,877 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:17.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,877 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,878 [shard 0] osd - peering_event(id=272, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,882 [shard 0] osd - peering_event(id=544, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,882 [shard 0] osd - peering_event(id=545, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,882 [shard 0] osd - peering_event(id=546, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,882 [shard 0] osd - peering_event(id=547, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:17.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,882 [shard 0] osd - peering_event(id=545, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:17.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,882 [shard 0] osd - peering_event(id=547, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:17.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - peering_event(id=546, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - peering_event(id=544, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:52:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569427490s 2022-03-29T17:52:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569427490s (sent 160.579177856s) 2022-03-29T17:52:17.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - peering_event(id=545, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:17.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.891 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:52:17.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569213867s 2022-03-29T17:52:17.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569213867s (sent 160.579452515s) 2022-03-29T17:52:17.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - peering_event(id=546, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:17.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:52:17.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569808960s 2022-03-29T17:52:17.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569808960s (sent 160.579666138s) 2022-03-29T17:52:17.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - peering_event(id=547, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:17.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,883 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:52:17.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.568969727s 2022-03-29T17:52:17.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.568969727s (sent 160.579895020s) 2022-03-29T17:52:17.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - peering_event(id=544, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:17.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - peering_event(id=545, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:17.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - peering_event(id=546, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:17.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - peering_event(id=544, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:17.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,884 [shard 0] osd - peering_event(id=547, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:17.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,905 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:52:17.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,905 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 160.559005737s))): start 2022-03-29T17:52:17.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 160.559005737s))): got map 18 2022-03-29T17:52:17.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 160.559005737s))): pg present 2022-03-29T17:52:17.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:52:17.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.496231079s ub 160.500320435s int 16.000000000s))): start 2022-03-29T17:52:17.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 160.559005737s) for pg: 2.3 2022-03-29T17:52:17.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 160.559005737s 2022-03-29T17:52:17.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 160.559005737s (sent 160.559005737s) 2022-03-29T17:52:17.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 160.559005737s))): submitting ctx 2022-03-29T17:52:17.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,906 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.496231079s ub 160.500320435s int 16.000000000s))): got map 18 2022-03-29T17:52:17.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.496231079s ub 160.500320435s int 16.000000000s))): pg present 2022-03-29T17:52:17.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.496231079s ub 160.500320435s int 16.000000000s) for pg: 2.0 2022-03-29T17:52:17.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 152.496231079s ub 160.500320435s int 16.000000000s) 2022-03-29T17:52:17.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 160.603363037s 2022-03-29T17:52:17.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 160.603363037s 2022-03-29T17:52:17.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.496231079s ub 160.500320435s int 16.000000000s))): submitting ctx 2022-03-29T17:52:17.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,907 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:52:17.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.506469727s ub 160.510345459s int 16.000000000s))): start 2022-03-29T17:52:17.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 160.559005737s))): complete 2022-03-29T17:52:17.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.496231079s ub 160.500320435s int 16.000000000s))): complete 2022-03-29T17:52:17.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.506469727s ub 160.510345459s int 16.000000000s))): got map 18 2022-03-29T17:52:17.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.506469727s ub 160.510345459s int 16.000000000s))): pg present 2022-03-29T17:52:17.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.506469727s ub 160.510345459s int 16.000000000s) for pg: 2.1 2022-03-29T17:52:17.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 152.506469727s ub 160.510345459s int 16.000000000s) 2022-03-29T17:52:17.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 160.604537964s 2022-03-29T17:52:17.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 160.604537964s 2022-03-29T17:52:17.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.506469727s ub 160.510345459s int 16.000000000s))): submitting ctx 2022-03-29T17:52:17.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,908 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:17,909 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 152.506469727s ub 160.510345459s int 16.000000000s))): complete 2022-03-29T17:52:17.939 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:17,933 [shard 0] alienstore - stat 2022-03-29T17:52:17.940 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:17,937 [shard 0] alienstore - stat 2022-03-29T17:52:17.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,946 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:52:17.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,946 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.500320435s))): start 2022-03-29T17:52:17.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,946 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.500320435s))): got map 18 2022-03-29T17:52:17.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.500320435s))): pg present 2022-03-29T17:52:17.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.500320435s) for pg: 2.0 2022-03-29T17:52:17.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 160.500320435s 2022-03-29T17:52:17.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 160.500320435s (sent 160.500320435s) 2022-03-29T17:52:17.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.500320435s))): submitting ctx 2022-03-29T17:52:17.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:52:17.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.510345459s))): start 2022-03-29T17:52:17.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.510345459s))): got map 18 2022-03-29T17:52:17.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,947 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.510345459s))): pg present 2022-03-29T17:52:17.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,948 [shard 0] osd - peering_event(id=273, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.500320435s))): complete 2022-03-29T17:52:17.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,948 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.510345459s) for pg: 2.1 2022-03-29T17:52:17.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,948 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 160.510345459s 2022-03-29T17:52:17.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,948 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 160.510345459s (sent 160.510345459s) 2022-03-29T17:52:17.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,948 [shard 0] osd - log is not dirty 2022-03-29T17:52:17.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,948 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.510345459s))): submitting ctx 2022-03-29T17:52:17.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,948 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:17.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:17,950 [shard 0] osd - peering_event(id=274, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 160.510345459s))): complete 2022-03-29T17:52:18.310 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:18,305 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is still pending... 2022-03-29T17:52:18.310 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,305 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:18.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,305 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=18145606070250940035 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:18.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,306 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:18.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,306 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=9111145872952205933 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:18.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,306 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:18.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,306 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:18.312 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,307 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.312 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:18,308 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:18.312 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,308 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.312 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:18,308 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:18.352 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:18,350 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is still pending... 2022-03-29T17:52:18.352 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,350 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:18.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,350 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=16200585162673444151 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:18.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:18.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=1328979418460197337 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:18.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:18.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,351 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:18.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,351 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.355 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:18,351 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:18.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,351 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.356 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:18,351 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:18.511 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,508 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:18.511 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,508 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:18.512 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,509 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.512 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:18,509 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:18.513 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,509 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.513 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:18,509 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:18.553 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,551 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:18.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,551 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:18.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,552 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.554 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:18,552 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:18.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,552 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.555 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:18,552 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:18.911 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,909 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:18.911 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,909 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:18.912 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,909 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.912 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:18,909 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:18.912 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,910 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.912 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:18,910 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:18.941 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:18,940 [shard 0] alienstore - stat 2022-03-29T17:52:18.945 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,944 [shard 0] alienstore - stat 2022-03-29T17:52:18.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,952 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:18.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,952 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:18.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,952 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.955 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:18,952 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:18.955 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:18,952 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:18.955 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:18,953 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:19.449 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,447 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:19.449 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,447 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:19.450 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,447 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:19.450 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:19,447 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] waiting 12.8 seconds ... 2022-03-29T17:52:19.450 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,447 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:19.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,710 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:19.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,710 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:19.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,710 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:19.712 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:19,710 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:19.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,710 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:19.713 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:19,710 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:19.755 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:19,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:19.756 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:19,753 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:19.757 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:19,753 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:19.757 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:19,753 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:19.757 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:19,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:19.758 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:19,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:19.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:19,947 [shard 0] alienstore - stat 2022-03-29T17:52:19.952 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:19,951 [shard 0] alienstore - stat 2022-03-29T17:52:20.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2022-03-29T17:52:20.063 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:20,061 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:20.063 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:20,061 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:20.064 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:20,061 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:20.064 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:20,061 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 12.8 seconds ... 2022-03-29T17:52:20.064 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:20,061 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:20.956 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:20,954 [shard 0] alienstore - stat 2022-03-29T17:52:20.959 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:20,957 [shard 0] alienstore - stat 2022-03-29T17:52:21.312 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:21,310 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:21.313 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:21,310 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:21.313 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:21,311 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:21.313 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:21,311 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 3.2 seconds ... 2022-03-29T17:52:21.313 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:21,311 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:21.314 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:21,311 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 3.2 seconds ... 2022-03-29T17:52:21.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:21,353 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:21.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:21,354 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:21.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:21,354 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:21.357 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:21,354 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 3.2 seconds ... 2022-03-29T17:52:21.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:21,354 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:21.357 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:21,354 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 3.2 seconds ... 2022-03-29T17:52:21.961 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:21,960 [shard 0] alienstore - stat 2022-03-29T17:52:21.966 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:21,964 [shard 0] alienstore - stat 2022-03-29T17:52:22.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,856 [shard 0] osd - peering_event(id=551, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:52:22.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,856 [shard 0] osd - peering_event(id=551, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:52:22.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,856 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:52:22.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,856 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 141.530502319s 2022-03-29T17:52:22.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,856 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 141.530502319s (sent 165.552413940s) 2022-03-29T17:52:22.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,856 [shard 0] osd - log is not dirty 2022-03-29T17:52:22.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,856 [shard 0] osd - peering_event(id=551, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:52:22.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,856 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:22.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:22,857 [shard 0] osd - peering_event(id=551, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:52:22.968 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:22,967 [shard 0] alienstore - stat 2022-03-29T17:52:22.973 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:22,971 [shard 0] alienstore - stat 2022-03-29T17:52:23.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,464 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,464 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,464 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1d 2022-03-29T17:52:23.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,464 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.095642090s 2022-03-29T17:52:23.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,464 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.095642090s (sent 166.097869873s) 2022-03-29T17:52:23.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,464 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,464 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,465 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.469 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,465 [shard 0] osd - peering_event(id=275, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.486 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,484 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,484 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,484 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:52:23.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,484 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.114166260s 2022-03-29T17:52:23.487 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,484 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.114166260s (sent 166.117797852s) 2022-03-29T17:52:23.488 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,484 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.488 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,484 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.488 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,484 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,485 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:52:23.489 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,485 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.114166260s ub 166.117797852s int 16.000000000s))): start 2022-03-29T17:52:23.489 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,486 [shard 0] osd - peering_event(id=276, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.491 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,486 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.114166260s ub 166.117797852s int 16.000000000s))): got map 18 2022-03-29T17:52:23.491 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,486 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.114166260s ub 166.117797852s int 16.000000000s))): pg present 2022-03-29T17:52:23.491 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,487 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.114166260s ub 166.117797852s int 16.000000000s) for pg: 3.8 2022-03-29T17:52:23.492 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,487 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.114166260s ub 166.117797852s int 16.000000000s) 2022-03-29T17:52:23.492 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,487 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.183013916s 2022-03-29T17:52:23.492 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,487 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.183013916s 2022-03-29T17:52:23.492 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,487 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.493 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,487 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.114166260s ub 166.117797852s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.493 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,487 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.493 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,487 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.114166260s ub 166.117797852s int 16.000000000s))): complete 2022-03-29T17:52:23.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.18 2022-03-29T17:52:23.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.121215820s 2022-03-29T17:52:23.500 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.121215820s (sent 166.127899170s) 2022-03-29T17:52:23.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,494 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:52:23.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 158.124221802s 2022-03-29T17:52:23.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 158.124221802s (sent 166.128143311s) 2022-03-29T17:52:23.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - peering_event(id=278, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - handle_peering_op on 3.8 from 1 2022-03-29T17:52:23.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,495 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.117797852s))): start 2022-03-29T17:52:23.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,496 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.117797852s))): got map 18 2022-03-29T17:52:23.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,496 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.117797852s))): pg present 2022-03-29T17:52:23.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,496 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.117797852s) for pg: 3.8 2022-03-29T17:52:23.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,496 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.117797852s 2022-03-29T17:52:23.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,496 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.117797852s (sent 166.117797852s) 2022-03-29T17:52:23.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,496 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,496 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.117797852s))): submitting ctx 2022-03-29T17:52:23.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,496 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,497 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:52:23.603 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,497 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.124221802s ub 166.128143311s int 16.000000000s))): start 2022-03-29T17:52:23.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,497 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.124221802s ub 166.128143311s int 16.000000000s))): got map 18 2022-03-29T17:52:23.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,497 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.124221802s ub 166.128143311s int 16.000000000s))): pg present 2022-03-29T17:52:23.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,497 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.124221802s ub 166.128143311s int 16.000000000s) for pg: 3.7 2022-03-29T17:52:23.604 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,498 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active proc_lease pg_lease(ru 158.124221802s ub 166.128143311s int 16.000000000s) 2022-03-29T17:52:23.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,498 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active no peer_clock_delta_lb -> ruub 166.193939209s 2022-03-29T17:52:23.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,498 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active proc_lease readable_until_ub now 166.193939209s 2022-03-29T17:52:23.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,498 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,498 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.124221802s ub 166.128143311s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,498 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,498 [shard 0] osd - peering_event(id=277, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,501 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.124221802s ub 166.128143311s int 16.000000000s))): complete 2022-03-29T17:52:23.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,501 [shard 0] osd - peering_event(id=279, detail=PeeringEvent(from=1 pgid=3.8 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.117797852s))): complete 2022-03-29T17:52:23.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,524 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,524 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,524 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:52:23.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.154312134s 2022-03-29T17:52:23.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.154312134s (sent 166.158020020s) 2022-03-29T17:52:23.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - handle_peering_op on 3.7 from 1 2022-03-29T17:52:23.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.128143311s))): start 2022-03-29T17:52:23.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - peering_event(id=280, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,525 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.128143311s))): got map 18 2022-03-29T17:52:23.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.128143311s))): pg present 2022-03-29T17:52:23.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.128143311s) for pg: 3.7 2022-03-29T17:52:23.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 166.128143311s 2022-03-29T17:52:23.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 166.128143311s (sent 166.128143311s) 2022-03-29T17:52:23.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.128143311s))): submitting ctx 2022-03-29T17:52:23.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:52:23.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,526 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.154312134s ub 166.158020020s int 16.000000000s))): start 2022-03-29T17:52:23.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,527 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.154312134s ub 166.158020020s int 16.000000000s))): got map 18 2022-03-29T17:52:23.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,527 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.154312134s ub 166.158020020s int 16.000000000s))): pg present 2022-03-29T17:52:23.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,527 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.154312134s ub 166.158020020s int 16.000000000s) for pg: 3.1e 2022-03-29T17:52:23.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,527 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.154312134s ub 166.158020020s int 16.000000000s) 2022-03-29T17:52:23.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,527 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.223815918s 2022-03-29T17:52:23.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,527 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.223815918s 2022-03-29T17:52:23.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,528 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,528 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.154312134s ub 166.158020020s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,528 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,528 [shard 0] osd - peering_event(id=281, detail=PeeringEvent(from=1 pgid=3.7 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.128143311s))): complete 2022-03-29T17:52:23.618 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,528 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.154312134s ub 166.158020020s int 16.000000000s))): complete 2022-03-29T17:52:23.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,544 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,544 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,545 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.e 2022-03-29T17:52:23.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,545 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.0 ruub 142.164947510s 2022-03-29T17:52:23.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,545 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 142.164947510s (sent 166.178085327s) 2022-03-29T17:52:23.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,545 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,545 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,545 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,545 [shard 0] osd - peering_event(id=282, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,554 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:52:23.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.184570312s 2022-03-29T17:52:23.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.184570312s (sent 166.188369751s) 2022-03-29T17:52:23.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - handle_peering_op on 3.1e from 1 2022-03-29T17:52:23.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,555 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.158020020s))): start 2022-03-29T17:52:23.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - peering_event(id=283, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.158020020s))): got map 18 2022-03-29T17:52:23.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.158020020s))): pg present 2022-03-29T17:52:23.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.158020020s) for pg: 3.1e 2022-03-29T17:52:23.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.158020020s 2022-03-29T17:52:23.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.158020020s (sent 166.158020020s) 2022-03-29T17:52:23.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.158020020s))): submitting ctx 2022-03-29T17:52:23.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,556 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,557 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:52:23.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,557 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.184570312s ub 166.188369751s int 16.000000000s))): start 2022-03-29T17:52:23.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,557 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.184570312s ub 166.188369751s int 16.000000000s))): got map 18 2022-03-29T17:52:23.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,557 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.184570312s ub 166.188369751s int 16.000000000s))): pg present 2022-03-29T17:52:23.630 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,558 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.184570312s ub 166.188369751s int 16.000000000s) for pg: 3.16 2022-03-29T17:52:23.630 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,558 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.184570312s ub 166.188369751s int 16.000000000s) 2022-03-29T17:52:23.630 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,558 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.254074097s 2022-03-29T17:52:23.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,558 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.254074097s 2022-03-29T17:52:23.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,558 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,558 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.184570312s ub 166.188369751s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.631 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,558 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,558 [shard 0] osd - peering_event(id=284, detail=PeeringEvent(from=1 pgid=3.1e sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.158020020s))): complete 2022-03-29T17:52:23.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,559 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.184570312s ub 166.188369751s int 16.000000000s))): complete 2022-03-29T17:52:23.633 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,560 [shard 0] osd - peering_event(id=556, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,560 [shard 0] osd - peering_event(id=556, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,560 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.19 2022-03-29T17:52:23.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,560 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.252090454s 2022-03-29T17:52:23.634 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,560 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.252090454s (sent 166.256790161s) 2022-03-29T17:52:23.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,561 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,561 [shard 0] osd - peering_event(id=556, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,561 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,561 [shard 0] osd - peering_event(id=556, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,565 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,565 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,565 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:52:23.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,565 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.194564819s 2022-03-29T17:52:23.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,565 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.194564819s (sent 166.198211670s) 2022-03-29T17:52:23.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,565 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,565 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,565 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - handle_peering_op on 3.16 from 1 2022-03-29T17:52:23.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.188369751s))): start 2022-03-29T17:52:23.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - peering_event(id=285, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.188369751s))): got map 18 2022-03-29T17:52:23.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.188369751s))): pg present 2022-03-29T17:52:23.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.188369751s) for pg: 3.16 2022-03-29T17:52:23.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.188369751s 2022-03-29T17:52:23.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.188369751s (sent 166.188369751s) 2022-03-29T17:52:23.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.188369751s))): submitting ctx 2022-03-29T17:52:23.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,566 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:52:23.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.252090454s ub 166.256790161s int 16.000000000s))): start 2022-03-29T17:52:23.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.252090454s ub 166.256790161s int 16.000000000s))): got map 18 2022-03-29T17:52:23.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.252090454s ub 166.256790161s int 16.000000000s))): pg present 2022-03-29T17:52:23.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.252090454s ub 166.256790161s int 16.000000000s) for pg: 3.19 2022-03-29T17:52:23.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.252090454s ub 166.256790161s int 16.000000000s) 2022-03-29T17:52:23.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.200515747s 2022-03-29T17:52:23.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.200515747s 2022-03-29T17:52:23.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.252090454s ub 166.256790161s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,567 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.646 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,568 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:52:23.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,568 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.194564819s ub 166.198211670s int 16.000000000s))): start 2022-03-29T17:52:23.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,568 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.194564819s ub 166.198211670s int 16.000000000s))): got map 18 2022-03-29T17:52:23.647 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,568 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.194564819s ub 166.198211670s int 16.000000000s))): pg present 2022-03-29T17:52:23.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,568 [shard 0] osd - peering_event(id=286, detail=PeeringEvent(from=1 pgid=3.16 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.188369751s))): complete 2022-03-29T17:52:23.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,569 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.194564819s ub 166.198211670s int 16.000000000s) for pg: 3.5 2022-03-29T17:52:23.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,569 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.194564819s ub 166.198211670s int 16.000000000s) 2022-03-29T17:52:23.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,569 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.265426636s 2022-03-29T17:52:23.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,569 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.265426636s 2022-03-29T17:52:23.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,569 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,569 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.194564819s ub 166.198211670s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,569 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,570 [shard 0] osd - handle_peering_op on 3.19 from 2 2022-03-29T17:52:23.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,570 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.256790161s))): start 2022-03-29T17:52:23.652 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,570 [shard 0] osd - handle_peering_op on 3.5 from 1 2022-03-29T17:52:23.652 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,570 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.198211670s))): start 2022-03-29T17:52:23.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,570 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.198211670s))): got map 18 2022-03-29T17:52:23.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,570 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.198211670s))): pg present 2022-03-29T17:52:23.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - peering_event(id=287, detail=PeeringEvent(from=1 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.252090454s ub 166.256790161s int 16.000000000s))): complete 2022-03-29T17:52:23.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.198211670s) for pg: 3.5 2022-03-29T17:52:23.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.198211670s 2022-03-29T17:52:23.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.198211670s (sent 166.198211670s) 2022-03-29T17:52:23.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.198211670s))): submitting ctx 2022-03-29T17:52:23.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.194564819s ub 166.198211670s int 16.000000000s))): complete 2022-03-29T17:52:23.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,571 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.256790161s))): got map 18 2022-03-29T17:52:23.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,572 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.256790161s))): pg present 2022-03-29T17:52:23.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,572 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.256790161s) for pg: 3.19 2022-03-29T17:52:23.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,572 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.256790161s 2022-03-29T17:52:23.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,572 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.256790161s (sent 166.256790161s) 2022-03-29T17:52:23.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,572 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,572 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.256790161s))): submitting ctx 2022-03-29T17:52:23.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,572 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,573 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.19 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.256790161s))): complete 2022-03-29T17:52:23.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,574 [shard 0] osd - peering_event(id=288, detail=PeeringEvent(from=1 pgid=3.5 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.198211670s))): complete 2022-03-29T17:52:23.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,595 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,595 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,595 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:52:23.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,595 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 158.224838257s 2022-03-29T17:52:23.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,595 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.224838257s (sent 166.228576660s) 2022-03-29T17:52:23.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,595 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,595 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,595 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,596 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:52:23.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,596 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.224838257s ub 166.228576660s int 16.000000000s))): start 2022-03-29T17:52:23.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,596 [shard 0] osd - peering_event(id=289, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,596 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.224838257s ub 166.228576660s int 16.000000000s))): got map 18 2022-03-29T17:52:23.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,597 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.224838257s ub 166.228576660s int 16.000000000s))): pg present 2022-03-29T17:52:23.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,597 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.224838257s ub 166.228576660s int 16.000000000s) for pg: 3.11 2022-03-29T17:52:23.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,597 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.224838257s ub 166.228576660s int 16.000000000s) 2022-03-29T17:52:23.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,597 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.293365479s 2022-03-29T17:52:23.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,597 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.293365479s 2022-03-29T17:52:23.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,597 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,597 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.224838257s ub 166.228576660s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,597 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,598 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 158.224838257s ub 166.228576660s int 16.000000000s))): complete 2022-03-29T17:52:23.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,612 [shard 0] osd - peering_event(id=560, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,612 [shard 0] osd - peering_event(id=560, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,612 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.14 2022-03-29T17:52:23.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,612 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.303329468s 2022-03-29T17:52:23.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,612 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.303329468s (sent 166.308303833s) 2022-03-29T17:52:23.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,612 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,612 [shard 0] osd - peering_event(id=560, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,612 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,613 [shard 0] osd - peering_event(id=560, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - peering_event(id=561, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - peering_event(id=562, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - peering_event(id=562, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - peering_event(id=561, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.671 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.0 2022-03-29T17:52:23.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.323913574s 2022-03-29T17:52:23.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.323913574s (sent 166.328613281s) 2022-03-29T17:52:23.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.672 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - peering_event(id=562, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,632 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1c 2022-03-29T17:52:23.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,633 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean recalc_readable_until peer osd.0 ruub 142.314071655s 2022-03-29T17:52:23.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,633 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean recalc_readable_until readable_until[_ub] 142.314071655s (sent 166.328933716s) 2022-03-29T17:52:23.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,633 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,633 [shard 0] osd - peering_event(id=561, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,633 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,633 [shard 0] osd - peering_event(id=562, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,633 [shard 0] osd - peering_event(id=561, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,637 [shard 0] osd - handle_peering_op on 3.11 from 1 2022-03-29T17:52:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,637 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.228576660s))): start 2022-03-29T17:52:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,637 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.228576660s))): got map 18 2022-03-29T17:52:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.228576660s))): pg present 2022-03-29T17:52:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.228576660s) for pg: 3.11 2022-03-29T17:52:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.228576660s 2022-03-29T17:52:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.228576660s (sent 166.228576660s) 2022-03-29T17:52:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.228576660s))): submitting ctx 2022-03-29T17:52:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:52:23.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.303329468s ub 166.308303833s int 16.000000000s))): start 2022-03-29T17:52:23.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.303329468s ub 166.308303833s int 16.000000000s))): got map 18 2022-03-29T17:52:23.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.303329468s ub 166.308303833s int 16.000000000s))): pg present 2022-03-29T17:52:23.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,638 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:52:23.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.323913574s ub 166.328613281s int 16.000000000s))): start 2022-03-29T17:52:23.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - peering_event(id=290, detail=PeeringEvent(from=1 pgid=3.11 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 166.228576660s))): complete 2022-03-29T17:52:23.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.323913574s ub 166.328613281s int 16.000000000s))): got map 18 2022-03-29T17:52:23.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.303329468s ub 166.308303833s int 16.000000000s) for pg: 3.14 2022-03-29T17:52:23.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.303329468s ub 166.308303833s int 16.000000000s) 2022-03-29T17:52:23.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.272476196s 2022-03-29T17:52:23.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.272476196s 2022-03-29T17:52:23.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.303329468s ub 166.308303833s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.323913574s ub 166.328613281s int 16.000000000s))): pg present 2022-03-29T17:52:23.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.323913574s ub 166.328613281s int 16.000000000s) for pg: 3.0 2022-03-29T17:52:23.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.323913574s ub 166.328613281s int 16.000000000s) 2022-03-29T17:52:23.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,639 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.272979736s 2022-03-29T17:52:23.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,640 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.272979736s 2022-03-29T17:52:23.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,640 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,640 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.323913574s ub 166.328613281s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,640 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,640 [shard 0] osd - peering_event(id=291, detail=PeeringEvent(from=1 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.303329468s ub 166.308303833s int 16.000000000s))): complete 2022-03-29T17:52:23.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,640 [shard 0] osd - peering_event(id=292, detail=PeeringEvent(from=1 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.323913574s ub 166.328613281s int 16.000000000s))): complete 2022-03-29T17:52:23.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - peering_event(id=563, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - peering_event(id=564, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.688 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - peering_event(id=564, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.689 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - peering_event(id=563, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.689 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.b 2022-03-29T17:52:23.689 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.336990356s 2022-03-29T17:52:23.689 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.336990356s (sent 166.348602295s) 2022-03-29T17:52:23.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - peering_event(id=564, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.690 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1a 2022-03-29T17:52:23.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,652 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until peer osd.2 ruub 158.343627930s 2022-03-29T17:52:23.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until readable_until[_ub] 158.343627930s (sent 166.348846436s) 2022-03-29T17:52:23.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - peering_event(id=563, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.691 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.693 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:52:23.693 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.343627930s ub 166.348846436s int 16.000000000s))): start 2022-03-29T17:52:23.693 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.343627930s ub 166.348846436s int 16.000000000s))): got map 18 2022-03-29T17:52:23.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.343627930s ub 166.348846436s int 16.000000000s))): pg present 2022-03-29T17:52:23.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,653 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.343627930s ub 166.348846436s int 16.000000000s) for pg: 3.1a 2022-03-29T17:52:23.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,654 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active proc_lease pg_lease(ru 158.343627930s ub 166.348846436s int 16.000000000s) 2022-03-29T17:52:23.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,654 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active no peer_clock_delta_lb -> ruub 166.287094116s 2022-03-29T17:52:23.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,654 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active proc_lease readable_until_ub now 166.287094116s 2022-03-29T17:52:23.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,654 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,654 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.343627930s ub 166.348846436s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,654 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,654 [shard 0] osd - peering_event(id=564, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,655 [shard 0] osd - handle_peering_op on 3.14 from 2 2022-03-29T17:52:23.696 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,655 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.308303833s))): start 2022-03-29T17:52:23.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,655 [shard 0] osd - peering_event(id=293, detail=PeeringEvent(from=1 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.343627930s ub 166.348846436s int 16.000000000s))): complete 2022-03-29T17:52:23.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.308303833s))): got map 18 2022-03-29T17:52:23.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.308303833s))): pg present 2022-03-29T17:52:23.705 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - peering_event(id=563, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - handle_peering_op on 3.0 from 2 2022-03-29T17:52:23.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.328613281s))): start 2022-03-29T17:52:23.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.308303833s) for pg: 3.14 2022-03-29T17:52:23.706 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.308303833s 2022-03-29T17:52:23.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.308303833s (sent 166.308303833s) 2022-03-29T17:52:23.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.308303833s))): submitting ctx 2022-03-29T17:52:23.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,656 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.707 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,657 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.328613281s))): got map 18 2022-03-29T17:52:23.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,657 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.328613281s))): pg present 2022-03-29T17:52:23.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,657 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.328613281s) for pg: 3.0 2022-03-29T17:52:23.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,657 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.328613281s 2022-03-29T17:52:23.708 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,657 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.328613281s (sent 166.328613281s) 2022-03-29T17:52:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,657 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,657 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.328613281s))): submitting ctx 2022-03-29T17:52:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,657 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - handle_peering_op on 3.1a from 2 2022-03-29T17:52:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.348846436s))): start 2022-03-29T17:52:23.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.348846436s))): got map 18 2022-03-29T17:52:23.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.348846436s))): pg present 2022-03-29T17:52:23.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=3.14 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.308303833s))): complete 2022-03-29T17:52:23.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.348846436s) for pg: 3.1a 2022-03-29T17:52:23.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until peer osd.2 ruub 166.348846436s 2022-03-29T17:52:23.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until readable_until[_ub] 166.348846436s (sent 166.348846436s) 2022-03-29T17:52:23.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.348846436s))): submitting ctx 2022-03-29T17:52:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,658 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,661 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.328613281s))): complete 2022-03-29T17:52:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,662 [shard 0] osd - peering_event(id=568, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,662 [shard 0] osd - peering_event(id=569, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,662 [shard 0] osd - peering_event(id=569, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,662 [shard 0] osd - peering_event(id=568, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,662 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.2 2022-03-29T17:52:23.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.353729248s 2022-03-29T17:52:23.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.353729248s (sent 166.358886719s) 2022-03-29T17:52:23.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - peering_event(id=568, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.714 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.13 2022-03-29T17:52:23.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.347534180s 2022-03-29T17:52:23.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.347534180s (sent 166.359237671s) 2022-03-29T17:52:23.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - peering_event(id=569, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.715 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,663 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.1a sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.348846436s))): complete 2022-03-29T17:52:23.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,666 [shard 0] osd - peering_event(id=568, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,666 [shard 0] osd - peering_event(id=569, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.716 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - peering_event(id=570, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - peering_event(id=571, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - peering_event(id=570, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - peering_event(id=571, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.4 2022-03-29T17:52:23.717 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 158.384033203s 2022-03-29T17:52:23.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 158.384033203s (sent 166.388748169s) 2022-03-29T17:52:23.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - peering_event(id=571, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.718 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,692 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,693 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.d 2022-03-29T17:52:23.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,693 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean recalc_readable_until peer osd.0 ruub 142.373336792s 2022-03-29T17:52:23.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,693 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean recalc_readable_until readable_until[_ub] 142.373336792s (sent 166.389053345s) 2022-03-29T17:52:23.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,693 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,693 [shard 0] osd - peering_event(id=570, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,693 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,693 [shard 0] osd - peering_event(id=571, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,694 [shard 0] osd - peering_event(id=570, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,697 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:52:23.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,697 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.353729248s ub 166.358886719s int 16.000000000s))): start 2022-03-29T17:52:23.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,697 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.353729248s ub 166.358886719s int 16.000000000s))): got map 18 2022-03-29T17:52:23.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,697 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.353729248s ub 166.358886719s int 16.000000000s))): pg present 2022-03-29T17:52:23.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.353729248s ub 166.358886719s int 16.000000000s) for pg: 3.2 2022-03-29T17:52:23.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.353729248s ub 166.358886719s int 16.000000000s) 2022-03-29T17:52:23.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.331222534s 2022-03-29T17:52:23.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.331222534s 2022-03-29T17:52:23.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.353729248s ub 166.358886719s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:52:23.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.384033203s ub 166.388748169s int 16.000000000s))): start 2022-03-29T17:52:23.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.384033203s ub 166.388748169s int 16.000000000s))): got map 18 2022-03-29T17:52:23.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,698 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.384033203s ub 166.388748169s int 16.000000000s))): pg present 2022-03-29T17:52:23.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,699 [shard 0] osd - peering_event(id=294, detail=PeeringEvent(from=1 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.353729248s ub 166.358886719s int 16.000000000s))): complete 2022-03-29T17:52:23.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,699 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.384033203s ub 166.388748169s int 16.000000000s) for pg: 3.4 2022-03-29T17:52:23.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,699 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 158.384033203s ub 166.388748169s int 16.000000000s) 2022-03-29T17:52:23.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,699 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 166.332168579s 2022-03-29T17:52:23.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,699 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 166.332168579s 2022-03-29T17:52:23.728 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,699 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.728 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,699 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.384033203s ub 166.388748169s int 16.000000000s))): submitting ctx 2022-03-29T17:52:23.729 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,699 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.729 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:23,700 [shard 0] osd - peering_event(id=295, detail=PeeringEvent(from=1 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 158.384033203s ub 166.388748169s int 16.000000000s))): complete 2022-03-29T17:52:23.730 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,712 [shard 0] osd - peering_event(id=572, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:23.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,712 [shard 0] osd - peering_event(id=572, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:23.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,712 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.10 2022-03-29T17:52:23.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,712 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 142.389938354s 2022-03-29T17:52:23.731 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,712 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 142.389938354s (sent 166.408737183s) 2022-03-29T17:52:23.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,712 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,712 [shard 0] osd - peering_event(id=572, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:23.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,712 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.732 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,713 [shard 0] osd - peering_event(id=572, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:23.739 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,738 [shard 0] osd - handle_peering_op on 3.2 from 2 2022-03-29T17:52:23.740 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,738 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.358886719s))): start 2022-03-29T17:52:23.742 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,738 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.358886719s))): got map 18 2022-03-29T17:52:23.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.358886719s))): pg present 2022-03-29T17:52:23.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.358886719s) for pg: 3.2 2022-03-29T17:52:23.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.358886719s 2022-03-29T17:52:23.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.358886719s (sent 166.358886719s) 2022-03-29T17:52:23.743 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.358886719s))): submitting ctx 2022-03-29T17:52:23.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - handle_peering_op on 3.4 from 2 2022-03-29T17:52:23.744 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,739 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.388748169s))): start 2022-03-29T17:52:23.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.388748169s))): got map 18 2022-03-29T17:52:23.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.388748169s))): pg present 2022-03-29T17:52:23.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.2 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.358886719s))): complete 2022-03-29T17:52:23.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.388748169s) for pg: 3.4 2022-03-29T17:52:23.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.388748169s 2022-03-29T17:52:23.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.388748169s (sent 166.388748169s) 2022-03-29T17:52:23.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - log is not dirty 2022-03-29T17:52:23.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.388748169s))): submitting ctx 2022-03-29T17:52:23.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,740 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:23.747 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:23,741 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.4 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 166.388748169s))): complete 2022-03-29T17:52:23.976 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:23,974 [shard 0] alienstore - stat 2022-03-29T17:52:23.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:23,977 [shard 0] alienstore - stat 2022-03-29T17:52:24.308 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:24,307 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is still pending... 2022-03-29T17:52:24.309 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,307 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:24.309 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,307 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=16724853838406304959 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:24.309 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,307 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:24.310 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,307 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=5579696548226707103 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:24.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,307 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:24.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,308 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:24.311 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,309 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.311 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:24,309 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:24.312 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,309 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.312 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:24,310 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:24.351 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:24,349 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is still pending... 2022-03-29T17:52:24.352 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,349 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:24.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,349 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=11823954523608641261 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:24.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,349 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:24.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=18340426048628765931 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:24.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,350 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:24.354 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:24.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.355 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:24,351 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:24.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,351 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.355 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:24,351 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:24.511 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,509 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:24.511 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,509 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.512 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:24,509 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:24.512 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,510 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:24.512 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,510 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.513 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:24,510 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:24.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,551 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:24.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,551 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:24.555 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,551 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.556 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:24,551 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:24.556 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,551 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.557 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:24,552 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:24.911 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,909 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:24.912 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,910 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.912 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:24,910 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:24.912 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,910 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:24.913 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,910 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.913 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:24,910 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:24.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,952 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:24.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,952 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:24.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,952 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.955 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:24,952 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:24.955 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,952 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:24.955 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:24,952 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:24.983 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:24,981 [shard 0] alienstore - stat 2022-03-29T17:52:24.985 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:24,983 [shard 0] alienstore - stat 2022-03-29T17:52:25.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2022-03-29T17:52:25.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:25,710 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:25.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:25,710 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:25.713 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:25,710 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:25.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:25,710 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:25.713 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:25,711 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:25.714 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:25,711 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:25.755 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:25,752 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:25.755 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:25,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:25.755 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:25,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:25.755 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:25,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:25.756 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:25,753 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:25.756 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:25,753 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:25.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,867 [shard 0] osd - peering_event(id=575, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:25.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,867 [shard 0] osd - peering_event(id=575, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:25.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,867 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.3 2022-03-29T17:52:25.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,867 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 160.559005737s 2022-03-29T17:52:25.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,867 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 160.559005737s (sent 168.563354492s) 2022-03-29T17:52:25.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,867 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,867 [shard 0] osd - peering_event(id=575, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:25.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,867 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,868 [shard 0] osd - handle_peering_op on 2.3 from 1 2022-03-29T17:52:25.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,868 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 160.559005737s ub 168.563354492s int 16.000000000s))): start 2022-03-29T17:52:25.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,868 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 160.559005737s ub 168.563354492s int 16.000000000s))): got map 18 2022-03-29T17:52:25.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,868 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 160.559005737s ub 168.563354492s int 16.000000000s))): pg present 2022-03-29T17:52:25.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,869 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 160.559005737s ub 168.563354492s int 16.000000000s) for pg: 2.3 2022-03-29T17:52:25.874 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,869 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 160.559005737s ub 168.563354492s int 16.000000000s) 2022-03-29T17:52:25.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,869 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 168.502120972s 2022-03-29T17:52:25.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,869 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 168.502120972s 2022-03-29T17:52:25.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,869 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,869 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 160.559005737s ub 168.563354492s int 16.000000000s))): submitting ctx 2022-03-29T17:52:25.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,869 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.876 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,869 [shard 0] osd - peering_event(id=575, detail=PeeringEvent(from=1 pgid=2.3 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:25.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:25.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:25.878 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - peering_event(id=296, detail=PeeringEvent(from=1 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.1 pg_lease(ru 160.559005737s ub 168.563354492s int 16.000000000s))): complete 2022-03-29T17:52:25.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:52:25.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 160.500320435s 2022-03-29T17:52:25.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 160.500320435s (sent 168.503524780s) 2022-03-29T17:52:25.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:25.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,870 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,871 [shard 0] osd - peering_event(id=297, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:25.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,880 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:25.882 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,880 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:25.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,880 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:52:25.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,880 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 160.510345459s 2022-03-29T17:52:25.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,880 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 160.510345459s (sent 168.513504028s) 2022-03-29T17:52:25.883 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,880 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,880 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:25.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,880 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,881 [shard 0] osd - peering_event(id=298, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:25.892 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - peering_event(id=576, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:25.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - peering_event(id=577, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:25.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - peering_event(id=578, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:25.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - peering_event(id=579, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:25.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - peering_event(id=576, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:25.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - peering_event(id=579, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:25.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - peering_event(id=577, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:25.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - peering_event(id=578, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:25.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.4 2022-03-29T17:52:25.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569427490s 2022-03-29T17:52:25.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,887 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569427490s (sent 168.583847046s) 2022-03-29T17:52:25.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - peering_event(id=576, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:25.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.5 2022-03-29T17:52:25.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569213867s 2022-03-29T17:52:25.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569213867s (sent 168.584045410s) 2022-03-29T17:52:25.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - peering_event(id=577, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:25.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.7 2022-03-29T17:52:25.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.568969727s 2022-03-29T17:52:25.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.568969727s (sent 168.584243774s) 2022-03-29T17:52:25.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - peering_event(id=579, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:25.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.6 2022-03-29T17:52:25.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.0 ruub 144.569808960s 2022-03-29T17:52:25.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 144.569808960s (sent 168.584487915s) 2022-03-29T17:52:25.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - peering_event(id=578, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:25.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,888 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,889 [shard 0] osd - peering_event(id=577, detail=PeeringEvent(from=1 pgid=2.5 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:25.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,890 [shard 0] osd - peering_event(id=579, detail=PeeringEvent(from=1 pgid=2.7 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:25.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,890 [shard 0] osd - peering_event(id=576, detail=PeeringEvent(from=1 pgid=2.4 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:25.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,890 [shard 0] osd - peering_event(id=578, detail=PeeringEvent(from=1 pgid=2.6 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:25.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,909 [shard 0] osd - handle_peering_op on 2.3 from 2 2022-03-29T17:52:25.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,909 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 168.563354492s))): start 2022-03-29T17:52:25.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,909 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 168.563354492s))): got map 18 2022-03-29T17:52:25.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 168.563354492s))): pg present 2022-03-29T17:52:25.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 168.563354492s) for pg: 2.3 2022-03-29T17:52:25.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 168.563354492s 2022-03-29T17:52:25.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.563354492s (sent 168.563354492s) 2022-03-29T17:52:25.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 168.563354492s))): submitting ctx 2022-03-29T17:52:25.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - handle_peering_op on 2.0 from 2 2022-03-29T17:52:25.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,910 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.500320435s ub 168.503524780s int 16.000000000s))): start 2022-03-29T17:52:25.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.500320435s ub 168.503524780s int 16.000000000s))): got map 18 2022-03-29T17:52:25.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.500320435s ub 168.503524780s int 16.000000000s))): pg present 2022-03-29T17:52:25.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - handle_peering_op on 2.1 from 2 2022-03-29T17:52:25.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.510345459s ub 168.513504028s int 16.000000000s))): start 2022-03-29T17:52:25.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=2.3 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.2 pg_lease_ack(ruub 168.563354492s))): complete 2022-03-29T17:52:25.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.500320435s ub 168.503524780s int 16.000000000s) for pg: 2.0 2022-03-29T17:52:25.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 160.500320435s ub 168.503524780s int 16.000000000s) 2022-03-29T17:52:25.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 168.607803345s 2022-03-29T17:52:25.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,911 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 168.607803345s 2022-03-29T17:52:25.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.500320435s ub 168.503524780s int 16.000000000s))): submitting ctx 2022-03-29T17:52:25.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.510345459s ub 168.513504028s int 16.000000000s))): got map 18 2022-03-29T17:52:25.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.510345459s ub 168.513504028s int 16.000000000s))): pg present 2022-03-29T17:52:25.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.510345459s ub 168.513504028s int 16.000000000s) for pg: 2.1 2022-03-29T17:52:25.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 160.510345459s ub 168.513504028s int 16.000000000s) 2022-03-29T17:52:25.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 168.608444214s 2022-03-29T17:52:25.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 168.608444214s 2022-03-29T17:52:25.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.510345459s ub 168.513504028s int 16.000000000s))): submitting ctx 2022-03-29T17:52:25.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,912 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.500320435s ub 168.503524780s int 16.000000000s))): complete 2022-03-29T17:52:25.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:25,913 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLease epoch 18 from osd.2 pg_lease(ru 160.510345459s ub 168.513504028s int 16.000000000s))): complete 2022-03-29T17:52:25.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,950 [shard 0] osd - handle_peering_op on 2.0 from 1 2022-03-29T17:52:25.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,950 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.503524780s))): start 2022-03-29T17:52:25.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,950 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.503524780s))): got map 18 2022-03-29T17:52:25.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,950 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.503524780s))): pg present 2022-03-29T17:52:25.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - handle_peering_op on 2.1 from 1 2022-03-29T17:52:25.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.513504028s))): start 2022-03-29T17:52:25.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.503524780s) for pg: 2.0 2022-03-29T17:52:25.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:52:25.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 168.503524780s) 2022-03-29T17:52:25.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.503524780s))): submitting ctx 2022-03-29T17:52:25.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.513504028s))): got map 18 2022-03-29T17:52:25.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.513504028s))): pg present 2022-03-29T17:52:25.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - do_peering_event handling epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.513504028s) for pg: 2.1 2022-03-29T17:52:25.958 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:52:25.958 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 168.513504028s) 2022-03-29T17:52:25.958 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,951 [shard 0] osd - log is not dirty 2022-03-29T17:52:25.958 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,952 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.513504028s))): submitting ctx 2022-03-29T17:52:25.958 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,952 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:25.959 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,952 [shard 0] osd - peering_event(id=299, detail=PeeringEvent(from=1 pgid=2.0 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.503524780s))): complete 2022-03-29T17:52:25.959 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:25,952 [shard 0] osd - peering_event(id=300, detail=PeeringEvent(from=1 pgid=2.1 sent=18 requested=18 evt=epoch_sent: 18 epoch_requested: 18 MLeaseAck epoch 18 from osd.1 pg_lease_ack(ruub 168.513504028s))): complete 2022-03-29T17:52:25.989 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:25,988 [shard 0] alienstore - stat 2022-03-29T17:52:25.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:25,991 [shard 0] alienstore - stat 2022-03-29T17:52:26.997 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:26,994 [shard 0] alienstore - stat 2022-03-29T17:52:26.999 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:26,998 [shard 0] alienstore - stat 2022-03-29T17:52:27.313 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:27,311 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:27.314 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:27,311 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:27.314 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:27,311 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:27.315 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:27,311 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 3.2 seconds ... 2022-03-29T17:52:27.315 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:27,312 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:27.316 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:27,312 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 3.2 seconds ... 2022-03-29T17:52:27.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:27,353 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:27.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:27,353 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:27.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:27,354 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:27.358 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:27,354 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 3.2 seconds ... 2022-03-29T17:52:27.358 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:27,354 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:27.359 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:27,354 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 3.2 seconds ... 2022-03-29T17:52:28.002 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:28,000 [shard 0] alienstore - stat 2022-03-29T17:52:28.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:28,004 [shard 0] alienstore - stat 2022-03-29T17:52:29.009 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:29,007 [shard 0] alienstore - stat 2022-03-29T17:52:29.013 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:29,011 [shard 0] alienstore - stat 2022-03-29T17:52:30.016 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,014 [shard 0] alienstore - stat 2022-03-29T17:52:30.020 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,018 [shard 0] alienstore - stat 2022-03-29T17:52:30.319 INFO:tasks.ceph.osd.1.smithi138.stderr:ERROR 2022-03-29 17:52:30,316 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.0 since back 2022-03-29 17:52:00 293370301 front 2022-03-29 17:52:00 293370301 (oldest deadline 2022-03-29 17:52:26 298247932) 2022-03-29T17:52:30.320 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,316 [shard 0] osd - add_pending: osd.0 failed for 30 2022-03-29T17:52:30.320 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:30,316 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is still pending... 2022-03-29T17:52:30.320 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,316 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:30.321 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,316 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=12681329976885935160 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:30.321 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,316 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:30.321 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,316 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=832631448393708905 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:30.321 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,317 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:30.322 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,318 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:30.323 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,319 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.323 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:30,319 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:30.323 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,319 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.323 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:30,319 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:30.352 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:52:30,350 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.0 since back 2022-03-29 17:52:00 348369180 front 2022-03-29 17:52:00 347369200 (oldest deadline 2022-03-29 17:52:26 348246913) 2022-03-29T17:52:30.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,350 [shard 0] osd - add_pending: osd.0 failed for 30 2022-03-29T17:52:30.353 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:30,350 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.0 is still pending... 2022-03-29T17:52:30.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,350 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:30.353 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,350 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6809/35577, peer_name=osd.0, cc=13784548795232185855 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:30.354 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:30.354 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,350 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6806/35577, peer_name=osd.0, cc=12559141330746127609 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:30.354 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,351 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:30.354 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,351 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:30.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,351 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.356 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:30,352 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.2 seconds ... 2022-03-29T17:52:30.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,352 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.356 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:30,352 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.2 seconds ... 2022-03-29T17:52:30.521 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,519 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:30.522 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,519 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:30.522 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,519 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.522 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:30,520 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:30.523 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,520 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.523 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:30,520 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:30.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,552 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:30.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,552 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:30.554 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,552 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.555 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:30,552 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.4 seconds ... 2022-03-29T17:52:30.555 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,552 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.555 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:30,552 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.4 seconds ... 2022-03-29T17:52:30.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2022-03-29T17:52:30.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,859 [shard 0] osd - peering_event(id=583, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): start 2022-03-29T17:52:30.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,859 [shard 0] osd - peering_event(id=583, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): pg present 2022-03-29T17:52:30.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,859 [shard 0] osd - do_peering_event handling epoch_sent: 8 epoch_requested: 8 RenewLease for pg: 1.0 2022-03-29T17:52:30.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,859 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until peer osd.0 ruub 141.530502319s 2022-03-29T17:52:30.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,859 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean recalc_readable_until readable_until[_ub] 141.530502319s (sent 173.555755615s) 2022-03-29T17:52:30.863 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,860 [shard 0] osd - log is not dirty 2022-03-29T17:52:30.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,860 [shard 0] osd - peering_event(id=583, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): submitting ctx 2022-03-29T17:52:30.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,860 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:30.864 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:30,860 [shard 0] osd - peering_event(id=583, detail=PeeringEvent(from=1 pgid=1.0 sent=8 requested=8 evt=epoch_sent: 8 epoch_requested: 8 RenewLease)): complete 2022-03-29T17:52:30.922 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,920 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:30.922 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,920 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:30.922 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,920 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.923 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:30,920 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:30.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:30,920 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.923 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:30,920 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:30.954 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,953 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:30.955 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,953 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:30.955 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,953 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.956 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:30,953 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 0.8 seconds ... 2022-03-29T17:52:30.956 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:30,953 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:30.956 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:30,953 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 0.8 seconds ... 2022-03-29T17:52:31.022 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,020 [shard 0] alienstore - stat 2022-03-29T17:52:31.026 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,024 [shard 0] alienstore - stat 2022-03-29T17:52:31.205 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,203 [shard 0] osd - handle_osd_map osd_map(18..19 src has 1..19) v4 2022-03-29T17:52:31.205 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,203 [shard 0] osd - handle_osd_map epochs [18..19], i have 18, src has [1..19] 2022-03-29T17:52:31.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,204 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:31.206 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,204 [shard 0] osd - handle_osd_map osd_map(19..19 src has 1..19) v4 2022-03-29T17:52:31.206 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,204 [shard 0] osd - handle_osd_map epochs [19..19], i have 19, src has [1..19] 2022-03-29T17:52:31.207 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,205 [shard 0] osd - handle_osd_map osd_map(18..19 src has 1..19) v4 2022-03-29T17:52:31.207 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,205 [shard 0] osd - handle_osd_map epochs [18..19], i have 18, src has [1..19] 2022-03-29T17:52:31.208 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,206 [shard 0] osd - osd.2: committed_osd_maps(19, 19) 2022-03-29T17:52:31.208 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,206 [shard 0] osd - osd.2: map e 19 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:31.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,206 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:31.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=84, detail=PGAdvanceMap(pg=2.0 from=18 to=19)): start 2022-03-29T17:52:31.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=85, detail=PGAdvanceMap(pg=2.1 from=18 to=19)): start 2022-03-29T17:52:31.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=86, detail=PGAdvanceMap(pg=2.3 from=18 to=19)): start 2022-03-29T17:52:31.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=87, detail=PGAdvanceMap(pg=3.0 from=18 to=19)): start 2022-03-29T17:52:31.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=88, detail=PGAdvanceMap(pg=3.1 from=18 to=19)): start 2022-03-29T17:52:31.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=89, detail=PGAdvanceMap(pg=3.2 from=18 to=19)): start 2022-03-29T17:52:31.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=90, detail=PGAdvanceMap(pg=3.3 from=18 to=19)): start 2022-03-29T17:52:31.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=91, detail=PGAdvanceMap(pg=3.4 from=18 to=19)): start 2022-03-29T17:52:31.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=92, detail=PGAdvanceMap(pg=3.5 from=18 to=19)): start 2022-03-29T17:52:31.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=93, detail=PGAdvanceMap(pg=3.7 from=18 to=19)): start 2022-03-29T17:52:31.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=94, detail=PGAdvanceMap(pg=3.8 from=18 to=19)): start 2022-03-29T17:52:31.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=95, detail=PGAdvanceMap(pg=3.9 from=18 to=19)): start 2022-03-29T17:52:31.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=96, detail=PGAdvanceMap(pg=3.e from=18 to=19)): start 2022-03-29T17:52:31.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=97, detail=PGAdvanceMap(pg=3.11 from=18 to=19)): start 2022-03-29T17:52:31.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=98, detail=PGAdvanceMap(pg=3.14 from=18 to=19)): start 2022-03-29T17:52:31.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=99, detail=PGAdvanceMap(pg=3.16 from=18 to=19)): start 2022-03-29T17:52:31.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=100, detail=PGAdvanceMap(pg=3.18 from=18 to=19)): start 2022-03-29T17:52:31.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=101, detail=PGAdvanceMap(pg=3.19 from=18 to=19)): start 2022-03-29T17:52:31.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=102, detail=PGAdvanceMap(pg=3.1a from=18 to=19)): start 2022-03-29T17:52:31.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=103, detail=PGAdvanceMap(pg=3.1d from=18 to=19)): start 2022-03-29T17:52:31.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_advance_map(id=104, detail=PGAdvanceMap(pg=3.1e from=18 to=19)): start 2022-03-29T17:52:31.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,207 [shard 0] osd - pg_epoch 18 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:31.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.248 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 31.754307 11 2022-03-29T17:51:59.459936+0000 2022-03-29T17:52:31.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.4537625, 1648576319.4599369 spent on 11 events 2022-03-29T17:52:31.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 31.754413 0 0.000000 2022-03-29T17:52:31.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.4537323, 0.0 spent on 0 events 2022-03-29T17:52:31.249 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started 31.763817 0 0.000000 2022-03-29T17:52:31.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - Exiting state: Started, entered at 1648576319.4443898, 0.0 spent on 0 events 2022-03-29T17:52:31.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.250 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 2}(0) acting {0, 2}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 2}(0) acting {0, 2}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.253 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,2 intervals=([14,18] acting 0,2)) 2022-03-29T17:52:31.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2}+{2} 2022-03-29T17:52:31.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.841766357s + 0.000000000s) 2022-03-29T17:52:31.255 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 2} -> {2}, acting {0, 2} -> {2}, acting_primary 0 -> 2, up_primary 0 -> 2, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - on_change, pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,208 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.256 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.000795 1 2022-03-29T17:52:31.208928+0000 2022-03-29T17:52:31.256 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2082753, 1648576351.2089283 spent on 1 events 2022-03-29T17:52:31.257 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.257 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.257 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.258 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.258 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.258 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000051 0 0.000000 2022-03-29T17:52:31.258 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Exiting state: Start, entered at 1648576351.2091577, 0.0 spent on 0 events 2022-03-29T17:52:31.259 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.259 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.259 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.259 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.259 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.260 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,2 2022-03-29T17:52:31.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,2 2022-03-29T17:52:31.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 2 down 0 blocked_by {} 2022-03-29T17:52:31.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:31.262 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000298 0 0.000000 2022-03-29T17:52:31.262 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2093332, 0.0 spent on 0 events 2022-03-29T17:52:31.262 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.263 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,209 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.2 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.2 with 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.263 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:52:31.264 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.264 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.264 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={2} async_recovery= 2022-03-29T17:52:31.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,209 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 2 2022-03-29T17:52:31.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={2} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.266 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000364 0 0.000000 2022-03-29T17:52:31.266 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2096872, 0.0 spent on 0 events 2022-03-29T17:52:31.266 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.266 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.267 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000070 0 0.000000 2022-03-29T17:52:31.267 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.210095, 0.0 spent on 0 events 2022-03-29T17:52:31.268 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.268 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.269 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,210 [shard 0] osd - send_alive want=19 up_thru_wanted=14 2022-03-29T17:52:31.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - send_alive up_thru_wanted=19 up_thru=14 2022-03-29T17:52:31.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,210 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: Active advmap 2022-03-29T17:52:31.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: Started advmap 2022-03-29T17:52:31.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 18 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:31.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {2} newacting {2} 2022-03-29T17:52:31.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {2} newacting {2} 2022-03-29T17:52:31.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 31.741739 10 0.003926 2022-03-29T17:52:31.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,211 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576319.4700906, 0.003926612 spent on 10 events 2022-03-29T17:52:31.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 31.913862 0 0.000000 2022-03-29T17:52:31.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,211 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576319.2980158, 0.0 spent on 0 events 2022-03-29T17:52:31.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,211 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary 32.873254 0 0.000000 2022-03-29T17:52:31.278 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,211 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576318.3386679, 0.0 spent on 0 events 2022-03-29T17:52:31.278 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started 32.873445 0 0.000000 2022-03-29T17:52:31.278 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,212 [shard 0] osd - Exiting state: Started, entered at 1648576318.3385386, 0.0 spent on 0 events 2022-03-29T17:52:31.278 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.279 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,212 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {2, 0}(2) acting {2, 0}(2)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {2, 0}(2) acting {2, 0}(2) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.281 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,2 intervals=([14,18] acting 0,2)) 2022-03-29T17:52:31.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2}+{2} 2022-03-29T17:52:31.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.845458984s + 0.000000000s) 2022-03-29T17:52:31.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {2, 0} -> {2}, acting {2, 0} -> {2}, acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - on_change, pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown {2, 0} -> {2}, replicas changed 2022-03-29T17:52:31.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean activate_map 2022-03-29T17:52:31.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: handle ActMap primary 2022-03-29T17:52:31.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean publish_stats_to_osd 18:11 2022-03-29T17:52:31.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean activate_map: Not dirtying info: last_persisted is 18 while current is 19 2022-03-29T17:52:31.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,212 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active state: Started advmap 2022-03-29T17:52:31.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 18 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:31.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {2} newacting {2} 2022-03-29T17:52:31.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {2} newacting {2} 2022-03-29T17:52:31.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.292 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,213 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 31.710447 10 0.002526 2022-03-29T17:52:31.292 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,213 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576319.5034573, 0.002526975 spent on 10 events 2022-03-29T17:52:31.293 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 31.910479 0 0.000000 2022-03-29T17:52:31.293 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576319.3035052, 0.0 spent on 0 events 2022-03-29T17:52:31.293 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary 32.844724 0 0.000000 2022-03-29T17:52:31.293 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576318.3693058, 0.0 spent on 0 events 2022-03-29T17:52:31.294 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started 32.844878 0 0.000000 2022-03-29T17:52:31.294 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - Exiting state: Started, entered at 1648576318.3691902, 0.0 spent on 0 events 2022-03-29T17:52:31.294 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.294 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {2, 0}(2) acting {2, 0}(2)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {2, 0}(2) acting {2, 0}(2) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.297 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,2 intervals=([14,18] acting 0,2)) 2022-03-29T17:52:31.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2}+{2} 2022-03-29T17:52:31.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.847595215s + 0.000000000s) 2022-03-29T17:52:31.299 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {2, 0} -> {2}, acting {2, 0} -> {2}, acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - on_change, pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown {2, 0} -> {2}, replicas changed 2022-03-29T17:52:31.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.300 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.002748 1 2022-03-29T17:52:31.212633+0000 2022-03-29T17:52:31.300 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - Exiting state: Reset, entered at 1648576351.212026, 1648576351.2126338 spent on 1 events 2022-03-29T17:52:31.301 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.301 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.301 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.302 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.302 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.302 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000048 0 0.000000 2022-03-29T17:52:31.303 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,214 [shard 0] osd - Exiting state: Start, entered at 1648576351.2148862, 0.0 spent on 0 events 2022-03-29T17:52:31.303 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.304 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.304 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.304 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.304 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.304 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,2 2022-03-29T17:52:31.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,2 2022-03-29T17:52:31.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 2 down 0 blocked_by {} 2022-03-29T17:52:31.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:11 2022-03-29T17:52:31.306 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000252 0 0.000000 2022-03-29T17:52:31.307 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2150614, 0.0 spent on 0 events 2022-03-29T17:52:31.307 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.307 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.2 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.2 with 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.308 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:52:31.308 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.308 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.308 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={2} async_recovery= 2022-03-29T17:52:31.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 2 2022-03-29T17:52:31.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={2} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.310 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000311 0 0.000000 2022-03-29T17:52:31.310 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2153554, 0.0 spent on 0 events 2022-03-29T17:52:31.310 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.310 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.311 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000057 0 0.000000 2022-03-29T17:52:31.311 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2157054, 0.0 spent on 0 events 2022-03-29T17:52:31.311 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.312 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,215 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,215 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.312 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:11 2022-03-29T17:52:31.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 18 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:31.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean new interval newup {2} newacting {2} 2022-03-29T17:52:31.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: Started advmap 2022-03-29T17:52:31.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean new interval newup {2} newacting {2} 2022-03-29T17:52:31.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.318 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean exit Started/Primary/Active/Clean 31.677322 10 0.002479 2022-03-29T17:52:31.318 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576319.5393136, 0.002479797 spent on 10 events 2022-03-29T17:52:31.318 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active exit Started/Primary/Active 31.886923 0 0.000000 2022-03-29T17:52:31.318 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576319.329759, 0.0 spent on 0 events 2022-03-29T17:52:31.319 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active exit Started/Primary 32.917956 0 0.000000 2022-03-29T17:52:31.319 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576318.2987678, 0.0 spent on 0 events 2022-03-29T17:52:31.319 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active exit Started 32.918105 0 0.000000 2022-03-29T17:52:31.319 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - Exiting state: Started, entered at 1648576318.2986567, 0.0 spent on 0 events 2022-03-29T17:52:31.319 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active enter Reset 2022-03-29T17:52:31.320 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,216 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active set_last_peering_reset 19 2022-03-29T17:52:31.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 lua=0'0 crt=18'1 mlcod 18'1 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 lua=0'0 crt=18'1 mlcod 18'1 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - client_request(id=168, detail=m=[osd_op(client.4179.0:90 3.a 3.fa1caca (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e19) v8]): start 2022-03-29T17:52:31.322 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - client_request(id=168, detail=m=[osd_op(client.4179.0:90 3.a 3.fa1caca (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e19) v8]): in repeat 2022-03-29T17:52:31.322 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,217 [shard 0] osd - evt epoch is 19, i have 18, will wait 2022-03-29T17:52:31.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,216 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 lua=0'0 crt=18'1 mlcod 18'1 active state: Reset advmap 2022-03-29T17:52:31.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 lua=0'0 crt=18'1 mlcod 18'1 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 lua=0'0 crt=18'1 mlcod 18'1 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,0] r=0 lpr=19 lua=0'0 crt=18'1 mlcod 18'1 active set_last_peering_reset 19 2022-03-29T17:52:31.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'1 mlcod 18'1 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {2, 0}(2) acting {2, 0}(2)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {2, 0}(2) acting {2, 0}(2) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.326 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'1 mlcod 18'1 active noting past ([14,18] all_participants=0,2 intervals=([14,18] acting 0,2)) 2022-03-29T17:52:31.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'1 mlcod 18'1 active on_new_interval 2022-03-29T17:52:31.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'1 mlcod 18'1 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2}+{2} 2022-03-29T17:52:31.327 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,217 [shard 0] osd - osdmap_subscribe(18) 2022-03-29T17:52:31.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,217 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'1 mlcod 18'1 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'1 mlcod 18'1 active init_hb_stamps now {} 2022-03-29T17:52:31.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'1 mlcod 18'1 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.851562500s + 0.000000000s) 2022-03-29T17:52:31.329 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'1 mlcod 18'1 active start_peering_interval up {2, 0} -> {2}, acting {2, 0} -> {2}, acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 18'1 unknown clear_primary_state 2022-03-29T17:52:31.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - on_change, pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown 2022-03-29T17:52:31.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown {2, 0} -> {2}, replicas changed 2022-03-29T17:52:31.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,218 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.332 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,219 [shard 0] osd - osd.1: committed_osd_maps(19, 19) 2022-03-29T17:52:31.332 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,219 [shard 0] osd - osd.1: map e 19 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:31.332 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - client_request(id=169, detail=m=[osd_op(client.4179.0:105 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e19) v8]): start 2022-03-29T17:52:31.333 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - client_request(id=169, detail=m=[osd_op(client.4179.0:105 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e19) v8]): in repeat 2022-03-29T17:52:31.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active activate_map 2022-03-29T17:52:31.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active activate_map: Not dirtying info: last_persisted is 17 while current is 19 2022-03-29T17:52:31.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,219 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.338 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,220 [shard 0] osd - handle_osd_map osd_map(18..19 src has 1..19) v4 2022-03-29T17:52:31.339 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,220 [shard 0] osd - handle_osd_map epochs [18..19], i have 19, src has [1..19] 2022-03-29T17:52:31.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_advance_map(id=188, detail=PGAdvanceMap(pg=1.0 from=18 to=19)): start 2022-03-29T17:52:31.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_advance_map(id=189, detail=PGAdvanceMap(pg=2.0 from=18 to=19)): start 2022-03-29T17:52:31.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_advance_map(id=190, detail=PGAdvanceMap(pg=2.1 from=18 to=19)): start 2022-03-29T17:52:31.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_advance_map(id=191, detail=PGAdvanceMap(pg=2.2 from=18 to=19)): start 2022-03-29T17:52:31.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:11 2022-03-29T17:52:31.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.344 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,220 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown exit Reset 0.004065 1 2022-03-29T17:52:31.218734+0000 2022-03-29T17:52:31.344 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,220 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2168002, 1648576351.218734 spent on 1 events 2022-03-29T17:52:31.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,220 [shard 0] osd - pg_advance_map(id=192, detail=PGAdvanceMap(pg=2.3 from=18 to=19)): start 2022-03-29T17:52:31.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=193, detail=PGAdvanceMap(pg=2.4 from=18 to=19)): start 2022-03-29T17:52:31.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=194, detail=PGAdvanceMap(pg=2.5 from=18 to=19)): start 2022-03-29T17:52:31.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=195, detail=PGAdvanceMap(pg=2.6 from=18 to=19)): start 2022-03-29T17:52:31.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=196, detail=PGAdvanceMap(pg=2.7 from=18 to=19)): start 2022-03-29T17:52:31.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=197, detail=PGAdvanceMap(pg=3.0 from=18 to=19)): start 2022-03-29T17:52:31.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=198, detail=PGAdvanceMap(pg=3.2 from=18 to=19)): start 2022-03-29T17:52:31.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=199, detail=PGAdvanceMap(pg=3.4 from=18 to=19)): start 2022-03-29T17:52:31.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=200, detail=PGAdvanceMap(pg=3.5 from=18 to=19)): start 2022-03-29T17:52:31.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=201, detail=PGAdvanceMap(pg=3.6 from=18 to=19)): start 2022-03-29T17:52:31.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=202, detail=PGAdvanceMap(pg=3.7 from=18 to=19)): start 2022-03-29T17:52:31.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=203, detail=PGAdvanceMap(pg=3.8 from=18 to=19)): start 2022-03-29T17:52:31.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=204, detail=PGAdvanceMap(pg=3.a from=18 to=19)): start 2022-03-29T17:52:31.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=205, detail=PGAdvanceMap(pg=3.b from=18 to=19)): start 2022-03-29T17:52:31.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=206, detail=PGAdvanceMap(pg=3.c from=18 to=19)): start 2022-03-29T17:52:31.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=207, detail=PGAdvanceMap(pg=3.d from=18 to=19)): start 2022-03-29T17:52:31.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=208, detail=PGAdvanceMap(pg=3.f from=18 to=19)): start 2022-03-29T17:52:31.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=209, detail=PGAdvanceMap(pg=3.10 from=18 to=19)): start 2022-03-29T17:52:31.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=210, detail=PGAdvanceMap(pg=3.11 from=18 to=19)): start 2022-03-29T17:52:31.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=211, detail=PGAdvanceMap(pg=3.12 from=18 to=19)): start 2022-03-29T17:52:31.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=212, detail=PGAdvanceMap(pg=3.13 from=18 to=19)): start 2022-03-29T17:52:31.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=213, detail=PGAdvanceMap(pg=3.14 from=18 to=19)): start 2022-03-29T17:52:31.354 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.354 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.355 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown exit Start 0.000061 0 0.000000 2022-03-29T17:52:31.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - Exiting state: Start, entered at 1648576351.2215314, 0.0 spent on 0 events 2022-03-29T17:52:31.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.356 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,221 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering build_prior all_probe 0,2 2022-03-29T17:52:31.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,2 2022-03-29T17:52:31.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering build_prior final: probe 2 down 0 blocked_by {} 2022-03-29T17:52:31.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering publish_stats_to_osd 18:11 2022-03-29T17:52:31.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,221 [shard 0] osd - pg_advance_map(id=214, detail=PGAdvanceMap(pg=3.15 from=18 to=19)): start 2022-03-29T17:52:31.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_advance_map(id=215, detail=PGAdvanceMap(pg=3.16 from=18 to=19)): start 2022-03-29T17:52:31.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_advance_map(id=216, detail=PGAdvanceMap(pg=3.17 from=18 to=19)): start 2022-03-29T17:52:31.361 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_advance_map(id=217, detail=PGAdvanceMap(pg=3.19 from=18 to=19)): start 2022-03-29T17:52:31.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_advance_map(id=218, detail=PGAdvanceMap(pg=3.1a from=18 to=19)): start 2022-03-29T17:52:31.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_advance_map(id=219, detail=PGAdvanceMap(pg=3.1b from=18 to=19)): start 2022-03-29T17:52:31.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_advance_map(id=220, detail=PGAdvanceMap(pg=3.1c from=18 to=19)): start 2022-03-29T17:52:31.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_advance_map(id=221, detail=PGAdvanceMap(pg=3.1e from=18 to=19)): start 2022-03-29T17:52:31.362 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_advance_map(id=222, detail=PGAdvanceMap(pg=3.1f from=18 to=19)): start 2022-03-29T17:52:31.363 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 18 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000239 0 0.000000 2022-03-29T17:52:31.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,222 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2216978, 0.0 spent on 0 events 2022-03-29T17:52:31.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,222 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering choose_acting all_info osd.2 3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering select_replicated_primary newest update on osd.2 with 3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.366 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:52:31.367 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.367 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.367 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering choose_async_recovery_replicated result want={2} async_recovery= 2022-03-29T17:52:31.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering acting_recovery_backfill is 2 2022-03-29T17:52:31.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering choose_acting want={2} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.368 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000334 0 0.000000 2022-03-29T17:52:31.369 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,222 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2225244, 0.0 spent on 0 events 2022-03-29T17:52:31.369 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.369 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,222 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.371 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.372 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.372 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 31.617244 12 2022-03-29T17:51:59.612813+0000 2022-03-29T17:52:31.372 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.6058252, 1648576319.6128135 spent on 12 events 2022-03-29T17:52:31.372 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 31.618369 0 0.000000 2022-03-29T17:52:31.373 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.604765, 0.0 spent on 0 events 2022-03-29T17:52:31.373 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started 31.629342 0 0.000000 2022-03-29T17:52:31.373 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - Exiting state: Started, entered at 1648576319.5938394, 0.0 spent on 0 events 2022-03-29T17:52:31.373 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.374 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.374 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.375 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,222 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000591 0 0.000000 2022-03-29T17:52:31.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2229004, 0.0 spent on 0 events 2022-03-29T17:52:31.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.378 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 18 while current is 19 2022-03-29T17:52:31.379 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.379 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,223 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.379 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.379 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.380 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.380 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.380 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,223 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.382 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.383 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.383 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.384 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.384 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.920120239s + 0.000000000s) 2022-03-29T17:52:31.384 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.384 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - on_change, pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.385 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:14 2022-03-29T17:52:31.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 18 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:31.390 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.391 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.001875 1 2022-03-29T17:52:31.224410+0000 2022-03-29T17:52:31.391 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2232244, 1648576351.2244105 spent on 1 events 2022-03-29T17:52:31.391 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.391 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.392 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.392 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.392 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.392 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000066 0 0.000000 2022-03-29T17:52:31.392 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Exiting state: Start, entered at 1648576351.2252095, 0.0 spent on 0 events 2022-03-29T17:52:31.393 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.393 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.393 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.393 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.394 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.394 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.394 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,224 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 31.784454 10 0.000838 2022-03-29T17:52:31.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.4411035, 0.000838519 spent on 10 events 2022-03-29T17:52:31.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 31.784528 0 0.000000 2022-03-29T17:52:31.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.4410763, 0.0 spent on 0 events 2022-03-29T17:52:31.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started 31.799832 0 0.000000 2022-03-29T17:52:31.399 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Exiting state: Started, entered at 1648576319.4258127, 0.0 spent on 0 events 2022-03-29T17:52:31.399 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.399 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,225 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:31.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.403 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.404 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:31.404 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000797 0 0.000000 2022-03-29T17:52:31.405 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,226 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2253714, 0.0 spent on 0 events 2022-03-29T17:52:31.405 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.405 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,226 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.405 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.406 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.406 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.406 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.406 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.407 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,225 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 2}(0) acting {0, 2}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 2}(0) acting {0, 2}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.409 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,2 intervals=([14,18] acting 0,2)) 2022-03-29T17:52:31.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2}+{2} 2022-03-29T17:52:31.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.859680176s + 0.000000000s) 2022-03-29T17:52:31.411 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 2} -> {2}, acting {0, 2} -> {2}, acting_primary 0 -> 2, up_primary 0 -> 2, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - on_change, pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 18 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:31.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,226 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.413 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.414 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.414 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000859 0 0.000000 2022-03-29T17:52:31.414 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.226212, 0.0 spent on 0 events 2022-03-29T17:52:31.414 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.415 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.415 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.415 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000062 0 0.000000 2022-03-29T17:52:31.415 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2271156, 0.0 spent on 0 events 2022-03-29T17:52:31.416 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.416 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.416 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active state: Started advmap 2022-03-29T17:52:31.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.420 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active exit Started/ReplicaActive/RepNotRecovering 31.809625 11 2022-03-29T17:51:59.426241+0000 2022-03-29T17:52:31.420 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.4179716, 1648576319.4262416 spent on 11 events 2022-03-29T17:52:31.420 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active exit Started/ReplicaActive 31.809727 0 0.000000 2022-03-29T17:52:31.420 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.4179235, 0.0 spent on 0 events 2022-03-29T17:52:31.421 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active exit Started 31.824737 0 0.000000 2022-03-29T17:52:31.421 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - Exiting state: Started, entered at 1648576319.402955, 0.0 spent on 0 events 2022-03-29T17:52:31.421 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.421 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,227 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.421 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active new interval newup {2} newacting {2} 2022-03-29T17:52:31.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.424 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,228 [shard 0] osd - send_alive want=19 up_thru_wanted=14 2022-03-29T17:52:31.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - send_alive up_thru_wanted=19 up_thru=14 2022-03-29T17:52:31.424 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 18 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.425 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,227 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,2] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 2}(0) acting {0, 2}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 2}(0) acting {0, 2}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.428 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active noting past ([14,18] all_participants=0,2 intervals=([14,18] acting 0,2)) 2022-03-29T17:52:31.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {2}+{2} 2022-03-29T17:52:31.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.861724854s + 0.000000000s) 2022-03-29T17:52:31.429 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active start_peering_interval up {0, 2} -> {2}, acting {0, 2} -> {2}, acting_primary 0 -> 2, up_primary 0 -> 2, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - on_change, pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown 2022-03-29T17:52:31.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.431 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 18 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 18 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.432 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 18 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.433 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,228 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.015388 1 2022-03-29T17:52:31.214729+0000 2022-03-29T17:52:31.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2141056, 1648576351.2147295 spent on 1 events 2022-03-29T17:52:31.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000052 0 0.000000 2022-03-29T17:52:31.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - Exiting state: Start, entered at 1648576351.2296147, 0.0 spent on 0 events 2022-03-29T17:52:31.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.439 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.439 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.439 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.439 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,229 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,2 2022-03-29T17:52:31.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,2 2022-03-29T17:52:31.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 2 down 0 blocked_by {} 2022-03-29T17:52:31.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.443 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.443 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 18 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.443 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.444 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.444 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 149.432870 47 0.007807 2022-03-29T17:52:31.445 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576201.7973044, 0.007807552 spent on 47 events 2022-03-29T17:52:31.445 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 149.489973 0 0.000000 2022-03-29T17:52:31.445 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576201.7402482, 0.0 spent on 0 events 2022-03-29T17:52:31.445 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary 150.485725 0 0.000000 2022-03-29T17:52:31.446 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576200.7445605, 0.0 spent on 0 events 2022-03-29T17:52:31.446 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started 150.485855 0 0.000000 2022-03-29T17:52:31.446 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Exiting state: Started, entered at 1648576200.7444663, 0.0 spent on 0 events 2022-03-29T17:52:31.446 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.447 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.447 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,229 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:11 2022-03-29T17:52:31.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000761 0 0.000000 2022-03-29T17:52:31.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2297583, 0.0 spent on 0 events 2022-03-29T17:52:31.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.2 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.2 with 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.451 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:52:31.452 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.452 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.452 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={2} async_recovery= 2022-03-29T17:52:31.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 2 2022-03-29T17:52:31.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={2} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.453 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000265 0 0.000000 2022-03-29T17:52:31.454 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.230581, 0.0 spent on 0 events 2022-03-29T17:52:31.454 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.454 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,230 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.456 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,230 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.456 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.457 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.457 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.457 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.457 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(11-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 12 interval(11-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.458 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.458 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active noting past ([11,18] all_participants=0,1 intervals=([11,18] acting 0,1)) 2022-03-29T17:52:31.458 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.458 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.927200317s + 0.000000000s) 2022-03-29T17:52:31.459 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.459 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.460 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - on_change, pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.460 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.462 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000053 0 0.000000 2022-03-29T17:52:31.462 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,231 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2308843, 0.0 spent on 0 events 2022-03-29T17:52:31.462 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.462 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,231 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.463 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,231 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,231 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 18 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.466 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.467 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.468 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 18 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.468 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.470 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:11 2022-03-29T17:52:31.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.471 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.472 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:11 2022-03-29T17:52:31.473 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.473 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.475 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.476 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.476 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 31.531630 11 0.002190 2022-03-29T17:52:31.476 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576319.7014508, 0.002190711 spent on 11 events 2022-03-29T17:52:31.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 31.866977 0 0.000000 2022-03-29T17:52:31.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576319.366159, 0.0 spent on 0 events 2022-03-29T17:52:31.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary 32.915716 0 0.000000 2022-03-29T17:52:31.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576318.3174677, 0.0 spent on 0 events 2022-03-29T17:52:31.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started 32.915864 0 0.000000 2022-03-29T17:52:31.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - Exiting state: Started, entered at 1648576318.317374, 0.0 spent on 0 events 2022-03-29T17:52:31.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,233 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.478 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,232 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 18 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,233 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.484 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.485 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.930175781s + 0.000000000s) 2022-03-29T17:52:31.487 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - on_change, pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.490 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown exit Reset 0.006786 1 2022-03-29T17:52:31.228865+0000 2022-03-29T17:52:31.490 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2277362, 1648576351.2288659 spent on 1 events 2022-03-29T17:52:31.491 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.491 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.491 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.491 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.492 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.492 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown exit Start 0.000053 0 0.000000 2022-03-29T17:52:31.492 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - Exiting state: Start, entered at 1648576351.2346091, 0.0 spent on 0 events 2022-03-29T17:52:31.492 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.492 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.493 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.493 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.493 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.493 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,234 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering build_prior all_probe 0,2 2022-03-29T17:52:31.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,2 2022-03-29T17:52:31.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering build_prior final: probe 2 down 0 blocked_by {} 2022-03-29T17:52:31.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,234 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 18 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active state: Started advmap 2022-03-29T17:52:31.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 18 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.498 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.499 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 149.434102 47 0.010616 2022-03-29T17:52:31.499 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576201.8012657, 0.010616249 spent on 47 events 2022-03-29T17:52:31.499 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 149.492013 0 0.000000 2022-03-29T17:52:31.499 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576201.7433968, 0.0 spent on 0 events 2022-03-29T17:52:31.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:31.502 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000761 0 0.000000 2022-03-29T17:52:31.502 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2347536, 0.0 spent on 0 events 2022-03-29T17:52:31.502 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.502 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering choose_acting all_info osd.2 3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering select_replicated_primary newest update on osd.2 with 3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.503 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:52:31.503 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.503 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.504 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering choose_async_recovery_replicated result want={2} async_recovery= 2022-03-29T17:52:31.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering acting_recovery_backfill is 2 2022-03-29T17:52:31.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering choose_acting want={2} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.505 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000258 0 0.000000 2022-03-29T17:52:31.505 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.235586, 0.0 spent on 0 events 2022-03-29T17:52:31.505 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.506 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.508 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,235 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary 150.495425 0 0.000000 2022-03-29T17:52:31.508 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576200.7405357, 0.0 spent on 0 events 2022-03-29T17:52:31.509 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started 150.495616 0 0.000000 2022-03-29T17:52:31.509 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - Exiting state: Started, entered at 1648576200.740411, 0.0 spent on 0 events 2022-03-29T17:52:31.509 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.509 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.510 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.511 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(11-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 12 interval(11-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.512 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active noting past ([11,18] all_participants=0,1 intervals=([11,18] acting 0,1)) 2022-03-29T17:52:31.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.512 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.513 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.513 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.514 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000057 0 0.000000 2022-03-29T17:52:31.514 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2358813, 0.0 spent on 0 events 2022-03-29T17:52:31.515 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.515 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 17 while current is 19 2022-03-29T17:52:31.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.516 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.932739258s + 0.000000000s) 2022-03-29T17:52:31.517 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,236 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:31.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:11 2022-03-29T17:52:31.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - on_change, pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.522 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 18 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.523 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: Started advmap 2022-03-29T17:52:31.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.524 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.524 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active+clean exit Started/Primary/Active/Clean 152.485621 54 0.010484 2022-03-29T17:52:31.524 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,237 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576198.7521944, 0.010484223 spent on 54 events 2022-03-29T17:52:31.525 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active exit Started/Primary/Active 152.533345 0 0.000000 2022-03-29T17:52:31.525 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,237 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576198.7045317, 0.0 spent on 0 events 2022-03-29T17:52:31.525 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active exit Started/Primary 153.535285 0 0.000000 2022-03-29T17:52:31.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,237 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.526 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.012321 1 2022-03-29T17:52:31.226778+0000 2022-03-29T17:52:31.526 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2256825, 1648576351.226778 spent on 1 events 2022-03-29T17:52:31.527 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.527 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.529 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,237 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576197.7026334, 0.0 spent on 0 events 2022-03-29T17:52:31.529 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active exit Started 153.535659 0 0.000000 2022-03-29T17:52:31.529 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Exiting state: Started, entered at 1648576197.7024782, 0.0 spent on 0 events 2022-03-29T17:52:31.529 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active enter Reset 2022-03-29T17:52:31.530 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=8 lua=0'0 crt=12'97 mlcod 12'97 active set_last_peering_reset 19 2022-03-29T17:52:31.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=19 lua=0'0 crt=12'97 mlcod 12'97 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=19 lua=0'0 crt=12'97 mlcod 12'97 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=19 lua=0'0 crt=12'97 mlcod 12'97 active state: Reset advmap 2022-03-29T17:52:31.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=19 lua=0'0 crt=12'97 mlcod 12'97 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=19 lua=0'0 crt=12'97 mlcod 12'97 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1,0] r=0 lpr=19 lua=0'0 crt=12'97 mlcod 12'97 active set_last_peering_reset 19 2022-03-29T17:52:31.534 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.534 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.534 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000443 0 0.000000 2022-03-29T17:52:31.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Exiting state: Start, entered at 1648576351.238092, 0.0 spent on 0 events 2022-03-29T17:52:31.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.536 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.536 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.536 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.536 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,2 2022-03-29T17:52:31.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,2 2022-03-29T17:52:31.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 2 down 0 blocked_by {} 2022-03-29T17:52:31.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:31.539 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000209 0 0.000000 2022-03-29T17:52:31.539 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.238652, 0.0 spent on 0 events 2022-03-29T17:52:31.539 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,238 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1] r=0 lpr=19 pi=[8,19)/1 lua=0'0 crt=12'97 mlcod 12'97 active start_peering_interval: check_new_interval output: check_new_interval interval(8-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 9 interval(8-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.541 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=8) [1] r=0 lpr=19 pi=[8,19)/1 lua=0'0 crt=12'97 mlcod 12'97 active noting past ([8,18] all_participants=0,1 intervals=([8,18] acting 0,1)) 2022-03-29T17:52:31.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 lua=0'0 crt=12'97 mlcod 12'97 active on_new_interval 2022-03-29T17:52:31.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 lua=0'0 crt=12'97 mlcod 12'97 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 lua=0'0 crt=12'97 mlcod 12'97 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.542 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 lua=0'0 crt=12'97 mlcod 12'97 active init_hb_stamps now {} 2022-03-29T17:52:31.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 lua=0'0 crt=12'97 mlcod 12'97 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.935043335s + 0.000000000s) 2022-03-29T17:52:31.543 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 lua=0'0 crt=12'97 mlcod 12'97 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 12'97 unknown clear_primary_state 2022-03-29T17:52:31.543 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - on_change, pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown 2022-03-29T17:52:31.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 18 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.544 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.547 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.2 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.2 with 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.547 INFO:tasks.ceph.osd.2.smithi138.stderr:up_primary: 2) selected as primary 2022-03-29T17:52:31.548 INFO:tasks.ceph.osd.2.smithi138.stderr:select_replicated_primary primary is osd.2 with 3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.548 INFO:tasks.ceph.osd.2.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.548 INFO:tasks.ceph.osd.2.smithi138.stderr: 2022-03-29T17:52:31.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={2} async_recovery= 2022-03-29T17:52:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 2 2022-03-29T17:52:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={2} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000820 0 0.000000 2022-03-29T17:52:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2389047, 0.0 spent on 0 events 2022-03-29T17:52:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.551 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000053 0 0.000000 2022-03-29T17:52:31.551 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2397685, 0.0 spent on 0 events 2022-03-29T17:52:31.551 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.551 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,239 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,239 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.554 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 31.708258 12 2022-03-29T17:51:59.548388+0000 2022-03-29T17:52:31.554 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.5317268, 1648576319.5483885 spent on 12 events 2022-03-29T17:52:31.555 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 31.708347 0 0.000000 2022-03-29T17:52:31.555 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.5316997, 0.0 spent on 0 events 2022-03-29T17:52:31.555 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started 31.727955 0 0.000000 2022-03-29T17:52:31.555 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - Exiting state: Started, entered at 1648576319.5121307, 0.0 spent on 0 events 2022-03-29T17:52:31.555 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.556 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.556 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.557 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.558 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.558 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,240 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,240 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.564 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.936965942s + 0.000000000s) 2022-03-29T17:52:31.564 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - on_change, pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.565 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 18 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.566 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.567 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:14 2022-03-29T17:52:31.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - peering_event(id=302, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:31.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,241 [shard 0] osd - peering_event(id=302, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:31.571 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,241 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 31.586091 11 0.001892 2022-03-29T17:52:31.572 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,241 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576319.6553423, 0.001892929 spent on 11 events 2022-03-29T17:52:31.572 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 31.900129 0 0.000000 2022-03-29T17:52:31.572 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,242 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576319.3418887, 0.0 spent on 0 events 2022-03-29T17:52:31.572 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary 32.882794 0 0.000000 2022-03-29T17:52:31.573 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,242 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576318.3592694, 0.0 spent on 0 events 2022-03-29T17:52:31.573 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started 32.882944 0 0.000000 2022-03-29T17:52:31.573 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,242 [shard 0] osd - Exiting state: Started, entered at 1648576318.3591545, 0.0 spent on 0 events 2022-03-29T17:52:31.573 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.574 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,242 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.574 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.575 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.576 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:31.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_advance_map(id=93, detail=PGAdvanceMap(pg=3.7 from=18 to=19)): complete 2022-03-29T17:52:31.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_advance_map(id=99, detail=PGAdvanceMap(pg=3.16 from=18 to=19)): complete 2022-03-29T17:52:31.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:31.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,242 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.579 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.580 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.939025879s + 0.000000000s) 2022-03-29T17:52:31.581 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - on_change, pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.581 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 18 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.582 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:31.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_advance_map(id=102, detail=PGAdvanceMap(pg=3.1a from=18 to=19)): complete 2022-03-29T17:52:31.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_advance_map(id=98, detail=PGAdvanceMap(pg=3.14 from=18 to=19)): complete 2022-03-29T17:52:31.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_advance_map(id=86, detail=PGAdvanceMap(pg=2.3 from=18 to=19)): complete 2022-03-29T17:52:31.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:12 2022-03-29T17:52:31.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.587 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.588 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:12 2022-03-29T17:52:31.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,243 [shard 0] osd - peering_event(id=304, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:31.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_advance_map(id=90, detail=PGAdvanceMap(pg=3.3 from=18 to=19)): complete 2022-03-29T17:52:31.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_advance_map(id=84, detail=PGAdvanceMap(pg=2.0 from=18 to=19)): complete 2022-03-29T17:52:31.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:31.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:31.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_advance_map(id=101, detail=PGAdvanceMap(pg=3.19 from=18 to=19)): complete 2022-03-29T17:52:31.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_advance_map(id=97, detail=PGAdvanceMap(pg=3.11 from=18 to=19)): complete 2022-03-29T17:52:31.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:31.594 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - pg_epoch 18 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 18'5 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 18'5 active state: Started advmap 2022-03-29T17:52:31.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 18'5 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.595 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 18'5 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.595 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 18'5 active exit Started/ReplicaActive/RepNotRecovering 31.808249 12 2022-03-29T17:51:59.447282+0000 2022-03-29T17:52:31.596 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,245 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.4368465, 1648576319.4472828 spent on 12 events 2022-03-29T17:52:31.596 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 18'5 active exit Started/ReplicaActive 31.808347 0 0.000000 2022-03-29T17:52:31.596 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,245 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.436797, 0.0 spent on 0 events 2022-03-29T17:52:31.596 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active exit Started 31.828143 0 0.000000 2022-03-29T17:52:31.596 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,245 [shard 0] osd - Exiting state: Started, entered at 1648576319.4170399, 0.0 spent on 0 events 2022-03-29T17:52:31.597 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.597 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,245 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.597 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,244 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.3 2022-03-29T17:52:31.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:31.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:31.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:31.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - peering_event(id=304, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:31.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.1d 2022-03-29T17:52:31.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:31.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - peering_event(id=302, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:31.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,245 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.605 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.606 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.942092896s + 0.000000000s) 2022-03-29T17:52:31.607 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'5 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.607 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - on_change, pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown 2022-03-29T17:52:31.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.608 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_advance_map(id=92, detail=PGAdvanceMap(pg=3.5 from=18 to=19)): complete 2022-03-29T17:52:31.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.18 2022-03-29T17:52:31.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:31.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:31.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,246 [shard 0] osd - pg_advance_map(id=91, detail=PGAdvanceMap(pg=3.4 from=18 to=19)): complete 2022-03-29T17:52:31.613 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.016625 1 2022-03-29T17:52:31.231995+0000 2022-03-29T17:52:31.613 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2303677, 1648576351.231995 spent on 1 events 2022-03-29T17:52:31.613 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.614 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.614 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.614 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.614 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.614 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000061 0 0.000000 2022-03-29T17:52:31.615 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - Exiting state: Start, entered at 1648576351.247106, 0.0 spent on 0 events 2022-03-29T17:52:31.615 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.615 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.615 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.616 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.616 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.616 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,247 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:11, acting: 0,1 2022-03-29T17:52:31.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.617 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_advance_map(id=94, detail=PGAdvanceMap(pg=3.8 from=18 to=19)): complete 2022-03-29T17:52:31.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_advance_map(id=103, detail=PGAdvanceMap(pg=3.1d from=18 to=19)): complete 2022-03-29T17:52:31.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_advance_map(id=100, detail=PGAdvanceMap(pg=3.18 from=18 to=19)): complete 2022-03-29T17:52:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.1 2022-03-29T17:52:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_advance_map(id=95, detail=PGAdvanceMap(pg=3.9 from=18 to=19)): complete 2022-03-29T17:52:31.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_advance_map(id=85, detail=PGAdvanceMap(pg=2.1 from=18 to=19)): complete 2022-03-29T17:52:31.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_advance_map(id=96, detail=PGAdvanceMap(pg=3.e from=18 to=19)): complete 2022-03-29T17:52:31.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_advance_map(id=89, detail=PGAdvanceMap(pg=3.2 from=18 to=19)): complete 2022-03-29T17:52:31.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_advance_map(id=87, detail=PGAdvanceMap(pg=3.0 from=18 to=19)): complete 2022-03-29T17:52:31.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,247 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:15 2022-03-29T17:52:31.624 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000772 0 0.000000 2022-03-29T17:52:31.625 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2472613, 0.0 spent on 0 events 2022-03-29T17:52:31.625 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.625 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.626 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.626 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.626 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.626 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.628 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.628 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000282 0 0.000000 2022-03-29T17:52:31.628 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.248078, 0.0 spent on 0 events 2022-03-29T17:52:31.628 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.628 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.629 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_advance_map(id=88, detail=PGAdvanceMap(pg=3.1 from=18 to=19)): complete 2022-03-29T17:52:31.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.e 2022-03-29T17:52:31.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:31.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - peering_event(id=304, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:31.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.632 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - pg_advance_map(id=104, detail=PGAdvanceMap(pg=3.1e from=18 to=19)): complete 2022-03-29T17:52:31.632 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - osd.2: now active 2022-03-29T17:52:31.633 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,248 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,248 [shard 0] osd - peering_event(id=301, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:31.634 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,249 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000570 0 0.000000 2022-03-29T17:52:31.635 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,249 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2483976, 0.0 spent on 0 events 2022-03-29T17:52:31.635 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,249 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.635 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,249 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.636 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,249 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - peering_event(id=302, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:31.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.9 2022-03-29T17:52:31.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:31.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:31.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.639 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,249 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - peering_event(id=303, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:31.648 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,249 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.649 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 18 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:31.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.650 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown exit Reset 0.012107 1 2022-03-29T17:52:31.239305+0000 2022-03-29T17:52:31.651 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2381825, 1648576351.2393053 spent on 1 events 2022-03-29T17:52:31.651 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.651 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.651 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown exit Start 0.000051 0 0.000000 2022-03-29T17:52:31.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Exiting state: Start, entered at 1648576351.2503722, 0.0 spent on 0 events 2022-03-29T17:52:31.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.653 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.653 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.653 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.653 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.654 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering build_prior maybe_rw interval:8, acting: 0,1 2022-03-29T17:52:31.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering publish_stats_to_osd 18:18 2022-03-29T17:52:31.655 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000214 0 0.000000 2022-03-29T17:52:31.656 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2505755, 0.0 spent on 0 events 2022-03-29T17:52:31.656 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.656 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,250 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.656 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering choose_acting all_info osd.1 1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) 2022-03-29T17:52:31.657 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) 2022-03-29T17:52:31.657 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.657 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) 2022-03-29T17:52:31.657 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.657 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,250 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.659 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000251 0 0.000000 2022-03-29T17:52:31.660 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.250831, 0.0 spent on 0 events 2022-03-29T17:52:31.660 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.660 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.660 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000059 0 0.000000 2022-03-29T17:52:31.661 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2511194, 0.0 spent on 0 events 2022-03-29T17:52:31.661 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.661 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.661 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.662 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.662 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 18 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active state: Started advmap 2022-03-29T17:52:31.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.663 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.663 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active exit Started/ReplicaActive/RepNotRecovering 31.786637 12 2022-03-29T17:51:59.479054+0000 2022-03-29T17:52:31.664 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.4649684, 1648576319.479054 spent on 12 events 2022-03-29T17:52:31.664 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 17'1 active exit Started/ReplicaActive 31.786717 0 0.000000 2022-03-29T17:52:31.664 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.4649384, 0.0 spent on 0 events 2022-03-29T17:52:31.664 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active exit Started 31.808081 0 0.000000 2022-03-29T17:52:31.665 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - Exiting state: Started, entered at 1648576319.4436116, 0.0 spent on 0 events 2022-03-29T17:52:31.665 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.665 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,251 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.665 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,251 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.667 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.948074341s + 0.000000000s) 2022-03-29T17:52:31.669 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=17'1 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - on_change, pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown 2022-03-29T17:52:31.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.670 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.671 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,252 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.672 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,252 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.672 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,252 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - peering_event(id=304, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:31.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - peering_event(id=305, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:31.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,252 [shard 0] osd - peering_event(id=306, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:31.813 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,252 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.019157 1 2022-03-29T17:52:31.234437+0000 2022-03-29T17:52:31.814 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2332861, 1648576351.2344375 spent on 1 events 2022-03-29T17:52:31.814 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.814 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.814 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.815 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.815 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.815 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000045 0 0.000000 2022-03-29T17:52:31.815 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Exiting state: Start, entered at 1648576351.2530663, 0.0 spent on 0 events 2022-03-29T17:52:31.816 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.816 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.816 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.816 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.816 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.817 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:31.817 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.818 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:12 2022-03-29T17:52:31.819 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000211 0 0.000000 2022-03-29T17:52:31.819 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2532032, 0.0 spent on 0 events 2022-03-29T17:52:31.819 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.819 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.819 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.820 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.820 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.820 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.820 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.821 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.821 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.822 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.822 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000246 0 0.000000 2022-03-29T17:52:31.822 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2534738, 0.0 spent on 0 events 2022-03-29T17:52:31.822 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.823 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.823 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.823 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000050 0 0.000000 2022-03-29T17:52:31.823 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2537603, 0.0 spent on 0 events 2022-03-29T17:52:31.824 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.824 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,253 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,253 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.824 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.825 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 18 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.825 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean state: Started advmap 2022-03-29T17:52:31.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.826 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.826 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active+clean exit Started/Primary/Active/Clean 31.624674 11 0.002105 2022-03-29T17:52:31.827 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576319.62958, 0.002105058 spent on 11 events 2022-03-29T17:52:31.827 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active exit Started/Primary/Active 31.935430 0 0.000000 2022-03-29T17:52:31.827 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576319.318865, 0.0 spent on 0 events 2022-03-29T17:52:31.827 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active exit Started/Primary 32.860451 0 0.000000 2022-03-29T17:52:31.827 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576318.393881, 0.0 spent on 0 events 2022-03-29T17:52:31.828 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active exit Started 32.860578 0 0.000000 2022-03-29T17:52:31.828 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - Exiting state: Started, entered at 1648576318.393787, 0.0 spent on 0 events 2022-03-29T17:52:31.828 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active enter Reset 2022-03-29T17:52:31.828 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'3 mlcod 18'3 active set_last_peering_reset 19 2022-03-29T17:52:31.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'3 mlcod 18'3 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'3 mlcod 18'3 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.829 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'3 mlcod 18'3 active state: Reset advmap 2022-03-29T17:52:31.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'3 mlcod 18'3 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'3 mlcod 18'3 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'3 mlcod 18'3 active set_last_peering_reset 19 2022-03-29T17:52:31.830 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'3 mlcod 18'3 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.831 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'3 mlcod 18'3 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'3 mlcod 18'3 active on_new_interval 2022-03-29T17:52:31.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'3 mlcod 18'3 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.831 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'3 mlcod 18'3 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'3 mlcod 18'3 active init_hb_stamps now {} 2022-03-29T17:52:31.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'3 mlcod 18'3 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.950683594s + 0.000000000s) 2022-03-29T17:52:31.832 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'3 mlcod 18'3 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.832 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 18'3 unknown clear_primary_state 2022-03-29T17:52:31.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - on_change, pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown 2022-03-29T17:52:31.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,254 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 18 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.833 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.834 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.835 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 149.461964 47 0.006538 2022-03-29T17:52:31.835 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576201.7932847, 0.006538301 spent on 47 events 2022-03-29T17:52:31.835 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 149.519736 0 0.000000 2022-03-29T17:52:31.835 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576201.7355566, 0.0 spent on 0 events 2022-03-29T17:52:31.835 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary 150.517551 0 0.000000 2022-03-29T17:52:31.836 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576200.7377777, 0.0 spent on 0 events 2022-03-29T17:52:31.836 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started 150.517682 0 0.000000 2022-03-29T17:52:31.836 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - Exiting state: Started, entered at 1648576200.7376778, 0.0 spent on 0 events 2022-03-29T17:52:31.836 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.837 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.837 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.838 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(11-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 12 interval(11-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.839 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active noting past ([11,18] all_participants=0,1 intervals=([11,18] acting 0,1)) 2022-03-29T17:52:31.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.839 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.951675415s + 0.000000000s) 2022-03-29T17:52:31.840 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.840 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - on_change, pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,255 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.841 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.842 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown exit Reset 0.001668 1 2022-03-29T17:52:31.254929+0000 2022-03-29T17:52:31.842 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Exiting state: Reset, entered at 1648576351.254404, 1648576351.254929 spent on 1 events 2022-03-29T17:52:31.842 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.842 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.842 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.843 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.843 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.843 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown exit Start 0.000046 0 0.000000 2022-03-29T17:52:31.843 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Exiting state: Start, entered at 1648576351.256156, 0.0 spent on 0 events 2022-03-29T17:52:31.844 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.844 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.844 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.844 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.844 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.845 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:31.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.845 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.846 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering publish_stats_to_osd 18:12 2022-03-29T17:52:31.846 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000238 0 0.000000 2022-03-29T17:52:31.847 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2563033, 0.0 spent on 0 events 2022-03-29T17:52:31.847 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.847 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering choose_acting all_info osd.1 3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.847 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.848 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.848 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.848 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.848 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.849 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.850 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000237 0 0.000000 2022-03-29T17:52:31.850 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2565787, 0.0 spent on 0 events 2022-03-29T17:52:31.850 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.850 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.851 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.851 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000053 0 0.000000 2022-03-29T17:52:31.851 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2568514, 0.0 spent on 0 events 2022-03-29T17:52:31.851 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.852 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,256 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 18 while current is 19 2022-03-29T17:52:31.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.852 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.852 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 18 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.853 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.854 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 31.770396 12 2022-03-29T17:51:59.493078+0000 2022-03-29T17:52:31.854 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.4869258, 1648576319.4930787 spent on 12 events 2022-03-29T17:52:31.854 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 31.771106 0 0.000000 2022-03-29T17:52:31.854 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.4862583, 0.0 spent on 0 events 2022-03-29T17:52:31.855 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started 31.790900 0 0.000000 2022-03-29T17:52:31.855 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - Exiting state: Started, entered at 1648576319.4665043, 0.0 spent on 0 events 2022-03-29T17:52:31.855 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.855 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.855 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.856 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.857 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.857 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.858 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.953720093s + 0.000000000s) 2022-03-29T17:52:31.859 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.859 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,257 [shard 0] osd - on_change, pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.860 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:12 2022-03-29T17:52:31.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.861 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 18 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.862 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.003022 1 2022-03-29T17:52:31.255924+0000 2022-03-29T17:52:31.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2553916, 1648576351.2559245 spent on 1 events 2022-03-29T17:52:31.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.863 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.864 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.864 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.864 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000044 0 0.000000 2022-03-29T17:52:31.864 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Exiting state: Start, entered at 1648576351.2585194, 0.0 spent on 0 events 2022-03-29T17:52:31.865 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.865 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.865 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.865 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.865 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.866 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:11, acting: 0,1 2022-03-29T17:52:31.866 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.867 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:15 2022-03-29T17:52:31.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000187 0 0.000000 2022-03-29T17:52:31.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2586703, 0.0 spent on 0 events 2022-03-29T17:52:31.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.868 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,258 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.868 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,258 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.869 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.869 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.869 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.869 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.870 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.870 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.871 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.871 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000247 0 0.000000 2022-03-29T17:52:31.871 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2588933, 0.0 spent on 0 events 2022-03-29T17:52:31.871 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.872 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.872 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.872 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000048 0 0.000000 2022-03-29T17:52:31.872 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2591763, 0.0 spent on 0 events 2022-03-29T17:52:31.872 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.873 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.873 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.873 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.874 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.875 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.875 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.019652 1 2022-03-29T17:52:31.241215+0000 2022-03-29T17:52:31.875 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2401223, 1648576351.2412155 spent on 1 events 2022-03-29T17:52:31.875 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.876 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.876 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.876 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.876 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.876 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000049 0 0.000000 2022-03-29T17:52:31.877 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Exiting state: Start, entered at 1648576351.2598476, 0.0 spent on 0 events 2022-03-29T17:52:31.877 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.877 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.877 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.878 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,259 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.878 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.878 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.878 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:31.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.879 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.880 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:31.880 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000225 0 0.000000 2022-03-29T17:52:31.880 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.259999, 0.0 spent on 0 events 2022-03-29T17:52:31.880 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.881 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.881 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.881 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.881 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.882 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.882 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.882 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.883 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.883 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000266 0 0.000000 2022-03-29T17:52:31.883 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2602959, 0.0 spent on 0 events 2022-03-29T17:52:31.884 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.884 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.884 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.884 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000060 0 0.000000 2022-03-29T17:52:31.884 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2605982, 0.0 spent on 0 events 2022-03-29T17:52:31.885 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.885 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.885 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.886 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,260 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,260 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.886 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:12 2022-03-29T17:52:31.887 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.888 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.889 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.889 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.019265 1 2022-03-29T17:52:31.243307+0000 2022-03-29T17:52:31.889 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Exiting state: Reset, entered at 1648576351.242133, 1648576351.2433076 spent on 1 events 2022-03-29T17:52:31.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.890 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000048 0 0.000000 2022-03-29T17:52:31.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Exiting state: Start, entered at 1648576351.2614877, 0.0 spent on 0 events 2022-03-29T17:52:31.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.891 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.892 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:31.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.893 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.894 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:12 2022-03-29T17:52:31.894 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000223 0 0.000000 2022-03-29T17:52:31.894 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2616296, 0.0 spent on 0 events 2022-03-29T17:52:31.895 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.895 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,261 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,261 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.895 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.895 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.896 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.896 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.896 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.896 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.897 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.897 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000256 0 0.000000 2022-03-29T17:52:31.898 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2618906, 0.0 spent on 0 events 2022-03-29T17:52:31.898 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.898 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.898 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.899 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000050 0 0.000000 2022-03-29T17:52:31.899 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2621808, 0.0 spent on 0 events 2022-03-29T17:52:31.899 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.899 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.899 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.900 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.900 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 18 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 18'3 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 18'3 active state: Started advmap 2022-03-29T17:52:31.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 18'3 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.901 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 18'3 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.901 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 18'3 active exit Started/ReplicaActive/RepNotRecovering 31.697527 11 0.001862 2022-03-29T17:52:31.902 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.5652618, 0.001862602 spent on 11 events 2022-03-29T17:52:31.902 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 18'3 active exit Started/ReplicaActive 31.697601 0 0.000000 2022-03-29T17:52:31.902 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.5652275, 0.0 spent on 0 events 2022-03-29T17:52:31.902 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active exit Started 31.722986 0 0.000000 2022-03-29T17:52:31.903 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - Exiting state: Started, entered at 1648576319.5398774, 0.0 spent on 0 events 2022-03-29T17:52:31.903 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.903 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,262 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.903 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,262 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.904 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.905 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.905 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.906 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.959213257s + 0.000000000s) 2022-03-29T17:52:31.907 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 luod=0'0 lua=0'0 crt=18'3 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - on_change, pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown 2022-03-29T17:52:31.907 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 18 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.908 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 31.691037 12 2022-03-29T17:51:59.576076+0000 2022-03-29T17:52:31.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.5725722, 1648576319.5760767 spent on 12 events 2022-03-29T17:52:31.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 31.691122 0 0.000000 2022-03-29T17:52:31.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.572546, 0.0 spent on 0 events 2022-03-29T17:52:31.909 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started 31.703215 0 0.000000 2022-03-29T17:52:31.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - Exiting state: Started, entered at 1648576319.5604866, 0.0 spent on 0 events 2022-03-29T17:52:31.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.910 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,263 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.912 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,263 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.959976196s + 0.000000000s) 2022-03-29T17:52:31.914 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - on_change, pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown exit Reset 0.012576 1 2022-03-29T17:52:31.252354+0000 2022-03-29T17:52:31.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2517278, 1648576351.2523546 spent on 1 events 2022-03-29T17:52:31.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.916 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown exit Start 0.000050 0 0.000000 2022-03-29T17:52:31.917 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Exiting state: Start, entered at 1648576351.2643735, 0.0 spent on 0 events 2022-03-29T17:52:31.917 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.917 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.917 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.918 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.918 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.918 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:31.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000225 0 0.000000 2022-03-29T17:52:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2645252, 0.0 spent on 0 events 2022-03-29T17:52:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,264 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering choose_acting all_info osd.1 3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.921 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.921 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.922 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.922 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,264 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.923 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000251 0 0.000000 2022-03-29T17:52:31.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2647862, 0.0 spent on 0 events 2022-03-29T17:52:31.924 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.924 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.924 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000048 0 0.000000 2022-03-29T17:52:31.924 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2650728, 0.0 spent on 0 events 2022-03-29T17:52:31.925 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.925 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 17 while current is 19 2022-03-29T17:52:31.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.926 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 18 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 18 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 18 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:31.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:31.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 18 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:31.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 31.676908 12 2022-03-29T17:51:59.598346+0000 2022-03-29T17:52:31.929 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.5889828, 1648576319.598347 spent on 12 events 2022-03-29T17:52:31.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 31.676979 0 0.000000 2022-03-29T17:52:31.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.5889547, 0.0 spent on 0 events 2022-03-29T17:52:31.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,265 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started 31.686522 0 0.000000 2022-03-29T17:52:31.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - Exiting state: Started, entered at 1648576319.579462, 0.0 spent on 0 events 2022-03-29T17:52:31.930 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.931 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.933 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.962280273s + 0.000000000s) 2022-03-29T17:52:31.934 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - on_change, pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 18 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.937 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 31.596082 11 0.001961 2022-03-29T17:52:31.937 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576319.6706355, 0.001961985 spent on 11 events 2022-03-29T17:52:31.937 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 31.920865 0 0.000000 2022-03-29T17:52:31.937 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576319.3458893, 0.0 spent on 0 events 2022-03-29T17:52:31.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started/Primary 32.884336 0 0.000000 2022-03-29T17:52:31.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576318.3824532, 0.0 spent on 0 events 2022-03-29T17:52:31.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active exit Started 32.884462 0 0.000000 2022-03-29T17:52:31.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - Exiting state: Started, entered at 1648576318.3823578, 0.0 spent on 0 events 2022-03-29T17:52:31.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.939 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,266 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,266 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.941 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.963104248s + 0.000000000s) 2022-03-29T17:52:31.943 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - on_change, pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 18 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:31.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Active advmap 2022-03-29T17:52:31.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Started advmap 2022-03-29T17:52:31.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:31.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.945 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown exit Reset 0.004715 1 2022-03-29T17:52:31.263437+0000 2022-03-29T17:52:31.945 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2629, 1648576351.2634377 spent on 1 events 2022-03-29T17:52:31.945 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.945 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.946 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.946 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.946 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.946 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown exit Start 0.000043 0 0.000000 2022-03-29T17:52:31.946 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - Exiting state: Start, entered at 1648576351.267695, 0.0 spent on 0 events 2022-03-29T17:52:31.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.948 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,267 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:31.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,267 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:31.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000278 0 0.000000 2022-03-29T17:52:31.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2678246, 0.0 spent on 0 events 2022-03-29T17:52:31.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.950 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering choose_acting all_info osd.1 3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.951 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.951 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.951 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.952 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000260 0 0.000000 2022-03-29T17:52:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2681415, 0.0 spent on 0 events 2022-03-29T17:52:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.954 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000068 0 0.000000 2022-03-29T17:52:31.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2684383, 0.0 spent on 0 events 2022-03-29T17:52:31.955 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.955 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 18 while current is 19 2022-03-29T17:52:31.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.956 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 18 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean state: Started advmap 2022-03-29T17:52:31.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.957 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active+clean exit Started/Primary/Active/Clean 31.582515 11 0.002260 2022-03-29T17:52:31.958 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576319.6863797, 0.002260369 spent on 11 events 2022-03-29T17:52:31.958 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active exit Started/Primary/Active 31.905910 0 0.000000 2022-03-29T17:52:31.958 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,268 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576319.3630512, 0.0 spent on 0 events 2022-03-29T17:52:31.958 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active exit Started/Primary 32.931483 0 0.000000 2022-03-29T17:52:31.959 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576318.337518, 0.0 spent on 0 events 2022-03-29T17:52:31.959 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active exit Started 32.933080 0 0.000000 2022-03-29T17:52:31.959 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - Exiting state: Started, entered at 1648576318.3359697, 0.0 spent on 0 events 2022-03-29T17:52:31.959 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active enter Reset 2022-03-29T17:52:31.959 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 lua=0'0 crt=18'4 mlcod 18'4 active set_last_peering_reset 19 2022-03-29T17:52:31.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'4 mlcod 18'4 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'4 mlcod 18'4 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'4 mlcod 18'4 active state: Reset advmap 2022-03-29T17:52:31.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'4 mlcod 18'4 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'4 mlcod 18'4 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=19 lua=0'0 crt=18'4 mlcod 18'4 active set_last_peering_reset 19 2022-03-29T17:52:31.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'4 mlcod 18'4 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.962 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'4 mlcod 18'4 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:31.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'4 mlcod 18'4 active on_new_interval 2022-03-29T17:52:31.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'4 mlcod 18'4 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'4 mlcod 18'4 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'4 mlcod 18'4 active init_hb_stamps now {} 2022-03-29T17:52:31.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'4 mlcod 18'4 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.965332031s + 0.000000000s) 2022-03-29T17:52:31.963 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 lua=0'0 crt=18'4 mlcod 18'4 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 18'4 unknown clear_primary_state 2022-03-29T17:52:31.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - on_change, pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown 2022-03-29T17:52:31.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:31.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:31.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:31.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:15 2022-03-29T17:52:31.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.966 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown exit Reset 0.024626 1 2022-03-29T17:52:31.246332+0000 2022-03-29T17:52:31.967 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2452192, 1648576351.2463322 spent on 1 events 2022-03-29T17:52:31.967 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.967 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.967 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.967 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.968 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.968 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown exit Start 0.000058 0 0.000000 2022-03-29T17:52:31.968 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,269 [shard 0] osd - Exiting state: Start, entered at 1648576351.2699194, 0.0 spent on 0 events 2022-03-29T17:52:31.968 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.968 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:31.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:31.971 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000230 0 0.000000 2022-03-29T17:52:31.971 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2700827, 0.0 spent on 0 events 2022-03-29T17:52:31.972 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.972 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering choose_acting all_info osd.1 3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.972 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.973 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:31.973 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.973 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.974 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000290 0 0.000000 2022-03-29T17:52:31.975 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2703512, 0.0 spent on 0 events 2022-03-29T17:52:31.975 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.975 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.976 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000056 0 0.000000 2022-03-29T17:52:31.976 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2706823, 0.0 spent on 0 events 2022-03-29T17:52:31.976 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.976 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 18 while current is 19 2022-03-29T17:52:31.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.977 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,270 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,270 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active activate_map 2022-03-29T17:52:31.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active activate_map: Not dirtying info: last_persisted is 18 while current is 19 2022-03-29T17:52:31.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 18 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:31.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap interval change, fast return 2022-03-29T17:52:31.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:31.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean new interval newup {1} newacting {1} 2022-03-29T17:52:31.980 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:31.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean exit Started/Primary/Active/Clean 149.484814 47 0.010509 2022-03-29T17:52:31.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - Exiting state: Started/Primary/Active/Clean, entered at 1648576201.7864933, 0.010509926 spent on 47 events 2022-03-29T17:52:31.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary/Active 149.537734 0 0.000000 2022-03-29T17:52:31.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - Exiting state: Started/Primary/Active, entered at 1648576201.733613, 0.0 spent on 0 events 2022-03-29T17:52:31.981 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started/Primary 150.524036 0 0.000000 2022-03-29T17:52:31.981 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - Exiting state: Started/Primary, entered at 1648576200.7473474, 0.0 spent on 0 events 2022-03-29T17:52:31.981 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active exit Started 150.524212 0 0.000000 2022-03-29T17:52:31.981 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - Exiting state: Started, entered at 1648576200.7472105, 0.0 spent on 0 events 2022-03-29T17:52:31.982 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:31.982 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:31.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=11 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:31.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:31.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:31.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:31.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:31.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:31.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,0] r=0 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:31.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(11-18 up {1, 0}(1) acting {1, 0}(1)) up_thru 14 up_from 7 last_epoch_clean 12 interval(11-18 up {1, 0}(1) acting {1, 0}(1) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:31.984 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active noting past ([11,18] all_participants=0,1 intervals=([11,18] acting 0,1)) 2022-03-29T17:52:31.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.967697144s + 0.000000000s) 2022-03-29T17:52:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {1, 0} -> {1}, acting {1, 0} -> {1}, acting_primary 1 -> 1, up_primary 1 -> 1, role 0 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - on_change, pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown {1, 0} -> {1}, replicas changed 2022-03-29T17:52:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,271 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:31.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.987 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.035988 1 2022-03-29T17:52:31.237601+0000 2022-03-29T17:52:31.987 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2360642, 1648576351.237601 spent on 1 events 2022-03-29T17:52:31.987 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:31.988 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Entering state: Started 2022-03-29T17:52:31.988 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:31.988 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Entering state: Start 2022-03-29T17:52:31.988 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:31.988 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000046 0 0.000000 2022-03-29T17:52:31.989 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Exiting state: Start, entered at 1648576351.2721324, 0.0 spent on 0 events 2022-03-29T17:52:31.989 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:31.989 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:31.989 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:31.989 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:31.990 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.990 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:31.990 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:31.990 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:11, acting: 0,1 2022-03-29T17:52:31.991 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:31.991 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:31.991 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:31.991 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:31.992 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:15 2022-03-29T17:52:31.992 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000208 0 0.000000 2022-03-29T17:52:31.992 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2722712, 0.0 spent on 0 events 2022-03-29T17:52:31.992 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:31.992 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:31.993 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.993 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.993 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:31.993 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:31.994 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:31.994 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:31.994 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:31.994 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:31.994 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:31.995 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:31.995 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:31.995 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000242 0 0.000000 2022-03-29T17:52:31.995 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2725332, 0.0 spent on 0 events 2022-03-29T17:52:31.996 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.996 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:31.996 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:31.996 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000050 0 0.000000 2022-03-29T17:52:31.996 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2728105, 0.0 spent on 0 events 2022-03-29T17:52:31.997 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.997 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,272 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:31.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:31.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,272 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.998 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:31.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:31.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:31.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:31.999 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - log is not dirty 2022-03-29T17:52:31.999 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:31.999 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:31.999 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.007325 1 2022-03-29T17:52:31.266495+0000 2022-03-29T17:52:31.999 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2660174, 1648576351.2664957 spent on 1 events 2022-03-29T17:52:32.000 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:32.000 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Entering state: Started 2022-03-29T17:52:32.000 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:32.000 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Entering state: Start 2022-03-29T17:52:32.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:32.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000057 0 0.000000 2022-03-29T17:52:32.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Exiting state: Start, entered at 1648576351.2734146, 0.0 spent on 0 events 2022-03-29T17:52:32.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:32.001 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:32.002 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:32.002 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:32.002 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.002 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:32.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:32.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:32.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:32.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:32.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:32.004 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000221 0 0.000000 2022-03-29T17:52:32.004 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2735639, 0.0 spent on 0 events 2022-03-29T17:52:32.005 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:32.005 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,273 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:32.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.005 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:32.006 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.006 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:32.006 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:32.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,273 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:32.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:32.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:32.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:32.007 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000240 0 0.000000 2022-03-29T17:52:32.008 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2738214, 0.0 spent on 0 events 2022-03-29T17:52:32.008 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.008 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:32.008 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000048 0 0.000000 2022-03-29T17:52:32.009 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2741017, 0.0 spent on 0 events 2022-03-29T17:52:32.009 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.009 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.010 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown activate_map 2022-03-29T17:52:32.011 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown exit Reset 0.005409 1 2022-03-29T17:52:31.269605+0000 2022-03-29T17:52:32.011 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2690816, 1648576351.2696052 spent on 1 events 2022-03-29T17:52:32.011 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown enter Started 2022-03-29T17:52:32.011 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Entering state: Started 2022-03-29T17:52:32.011 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown enter Start 2022-03-29T17:52:32.012 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Entering state: Start 2022-03-29T17:52:32.012 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:32.012 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown exit Start 0.000050 0 0.000000 2022-03-29T17:52:32.012 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Exiting state: Start, entered at 1648576351.2745628, 0.0 spent on 0 events 2022-03-29T17:52:32.013 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:32.013 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:32.013 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:32.013 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:32.013 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.014 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.014 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:32.014 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:32.014 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:32.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:32.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:32.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering publish_stats_to_osd 18:12 2022-03-29T17:52:32.015 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000202 0 0.000000 2022-03-29T17:52:32.016 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2747192, 0.0 spent on 0 events 2022-03-29T17:52:32.016 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:32.016 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,274 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:32.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering choose_acting all_info osd.1 3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.017 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:32.017 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.017 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:32.018 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:32.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:32.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:32.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:32.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:32.019 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000225 0 0.000000 2022-03-29T17:52:32.019 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2749727, 0.0 spent on 0 events 2022-03-29T17:52:32.019 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.020 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:32.020 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000072 0 0.000000 2022-03-29T17:52:32.020 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2752323, 0.0 spent on 0 events 2022-03-29T17:52:32.020 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.021 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 18 while current is 19 2022-03-29T17:52:32.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.022 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 18 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:32.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:32.023 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 149.496535 33 2022-03-29T17:50:01.789267+0000 2022-03-29T17:52:32.023 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576201.779118, 1648576201.7892673 spent on 33 events 2022-03-29T17:52:32.023 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 149.496612 0 0.000000 2022-03-29T17:52:32.024 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576201.7790802, 0.0 spent on 0 events 2022-03-29T17:52:32.024 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active exit Started 149.500511 0 0.000000 2022-03-29T17:52:32.024 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - Exiting state: Started, entered at 1648576201.7752137, 0.0 spent on 0 events 2022-03-29T17:52:32.024 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:32.025 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,275 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:32.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:32.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:32.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:32.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:32.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:32.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:32.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:32.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:32.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,275 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(11-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 12 interval(11-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:32.027 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active noting past ([11,18] all_participants=0,1 intervals=([11,18] acting 0,1)) 2022-03-29T17:52:32.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:32.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:32.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:32.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:32.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.972000122s + 0.000000000s) 2022-03-29T17:52:32.028 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:32.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:32.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - on_change, pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:32.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:32.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:32.030 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.018967 1 2022-03-29T17:52:31.258013+0000 2022-03-29T17:52:32.030 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2574427, 1648576351.2580137 spent on 1 events 2022-03-29T17:52:32.031 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:32.031 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Entering state: Started 2022-03-29T17:52:32.031 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:32.031 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Entering state: Start 2022-03-29T17:52:32.032 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:32.032 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000040 0 0.000000 2022-03-29T17:52:32.032 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Exiting state: Start, entered at 1648576351.276502, 0.0 spent on 0 events 2022-03-29T17:52:32.032 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:32.033 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:32.033 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:32.033 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:32.033 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.033 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:32.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:32.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:32.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:32.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:32.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:32.035 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000183 0 0.000000 2022-03-29T17:52:32.035 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2766297, 0.0 spent on 0 events 2022-03-29T17:52:32.036 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:32.036 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,276 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:32.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,276 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.037 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:32.037 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.037 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:32.037 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:32.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:32.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:32.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:32.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:32.039 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000239 0 0.000000 2022-03-29T17:52:32.039 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2768703, 0.0 spent on 0 events 2022-03-29T17:52:32.039 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.039 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:32.040 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000049 0 0.000000 2022-03-29T17:52:32.040 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.27716, 0.0 spent on 0 events 2022-03-29T17:52:32.040 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.040 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.041 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 18 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:32.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: should_restart_peering, transitioning to Reset 2022-03-29T17:52:32.043 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive/RepNotRecovering 31.764396 12 2022-03-29T17:51:59.519861+0000 2022-03-29T17:52:32.043 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - Exiting state: Started/ReplicaActive/RepNotRecovering, entered at 1648576319.5131972, 1648576319.5198615 spent on 12 events 2022-03-29T17:52:32.043 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started/ReplicaActive 31.764463 0 0.000000 2022-03-29T17:52:32.044 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - Exiting state: Started/ReplicaActive, entered at 1648576319.5131695, 0.0 spent on 0 events 2022-03-29T17:52:32.044 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active exit Started 31.785783 0 0.000000 2022-03-29T17:52:32.044 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - Exiting state: Started, entered at 1648576319.4918883, 0.0 spent on 0 events 2022-03-29T17:52:32.044 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active enter Reset 2022-03-29T17:52:32.044 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,277 [shard 0] osd - Entering state: Reset 2022-03-29T17:52:32.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:32.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Clearing blocked outgoing recovery messages 2022-03-29T17:52:32.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - PG::try_flush_or_schedule_async: do_transaction... 2022-03-29T17:52:32.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active Beginning to block outgoing recovery messages 2022-03-29T17:52:32.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: Reset advmap 2022-03-29T17:52:32.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active new interval newup {1} newacting {1} 2022-03-29T17:52:32.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active state: should restart peering, calling start_peering_interval again 2022-03-29T17:52:32.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [0,1] r=1 lpr=19 crt=0'0 mlcod 0'0 active set_last_peering_reset 19 2022-03-29T17:52:32.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval: check_new_interval output: check_new_interval interval(14-18 up {0, 1}(0) acting {0, 1}(0)) up_thru 14 up_from 7 last_epoch_clean 15 interval(14-18 up {0, 1}(0) acting {0, 1}(0) maybe_went_rw) : primary up 7-14 includes interval 2022-03-29T17:52:32.047 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active noting past ([14,18] all_participants=0,1 intervals=([14,18] acting 0,1)) 2022-03-29T17:52:32.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval 2022-03-29T17:52:32.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval upacting_features 0x3f01cfbb7ffdffff from {1}+{1} 2022-03-29T17:52:32.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,277 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2022-03-29T17:52:32.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active init_hb_stamps now {} 2022-03-29T17:52:32.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active on_new_interval prior_readable_until_ub 0.000000000s (mnow 157.973937988s + 0.000000000s) 2022-03-29T17:52:32.048 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active start_peering_interval up {0, 1} -> {1}, acting {0, 1} -> {1}, acting_primary 0 -> 1, up_primary 0 -> 1, role 1 -> 0, features acting 4540138303579357183 upacting 4540138303579357183 2022-03-29T17:52:32.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown clear_primary_state 2022-03-29T17:52:32.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - on_change, pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown 2022-03-29T17:52:32.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown check_recovery_sources no source osds () went down 2022-03-29T17:52:32.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:32.050 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.000651 1 2022-03-29T17:52:31.278140+0000 2022-03-29T17:52:32.050 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2777028, 1648576351.27814 spent on 1 events 2022-03-29T17:52:32.051 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:32.051 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Entering state: Started 2022-03-29T17:52:32.051 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:32.051 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Entering state: Start 2022-03-29T17:52:32.051 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:32.052 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000099 0 0.000000 2022-03-29T17:52:32.052 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Exiting state: Start, entered at 1648576351.2784386, 0.0 spent on 0 events 2022-03-29T17:52:32.052 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:32.052 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:32.053 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:32.053 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:32.053 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.053 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:32.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:32.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:32.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:32.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:32.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:32.055 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000201 0 0.000000 2022-03-29T17:52:32.055 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2786205, 0.0 spent on 0 events 2022-03-29T17:52:32.055 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:32.056 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,278 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:32.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,278 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.056 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:32.057 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.057 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:32.057 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:32.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:32.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:32.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:32.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:32.058 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000264 0 0.000000 2022-03-29T17:52:32.059 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2788565, 0.0 spent on 0 events 2022-03-29T17:52:32.059 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.059 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:32.059 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000047 0 0.000000 2022-03-29T17:52:32.060 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2791557, 0.0 spent on 0 events 2022-03-29T17:52:32.060 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.060 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.061 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:32.062 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.012637 1 2022-03-29T17:52:31.267367+0000 2022-03-29T17:52:32.062 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Exiting state: Reset, entered at 1648576351.266853, 1648576351.2673674 spent on 1 events 2022-03-29T17:52:32.062 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:32.062 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Entering state: Started 2022-03-29T17:52:32.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:32.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Entering state: Start 2022-03-29T17:52:32.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:32.063 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000042 0 0.000000 2022-03-29T17:52:32.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Exiting state: Start, entered at 1648576351.279562, 0.0 spent on 0 events 2022-03-29T17:52:32.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:32.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:32.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:32.064 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:32.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.065 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:32.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:32.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:32.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:32.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:32.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:12 2022-03-29T17:52:32.067 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000184 0 0.000000 2022-03-29T17:52:32.067 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.279691, 0.0 spent on 0 events 2022-03-29T17:52:32.067 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:32.067 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,279 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:32.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,279 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.068 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:32.068 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.068 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:32.069 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:32.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:32.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:32.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:32.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:32.070 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000244 0 0.000000 2022-03-29T17:52:32.070 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.279911, 0.0 spent on 0 events 2022-03-29T17:52:32.070 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.071 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:32.071 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000058 0 0.000000 2022-03-29T17:52:32.071 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2801936, 0.0 spent on 0 events 2022-03-29T17:52:32.072 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.072 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.073 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:32.073 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.004790 1 2022-03-29T17:52:31.276213+0000 2022-03-29T17:52:32.074 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2757561, 1648576351.2762134 spent on 1 events 2022-03-29T17:52:32.074 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:32.074 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Entering state: Started 2022-03-29T17:52:32.074 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:32.074 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Entering state: Start 2022-03-29T17:52:32.075 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:32.075 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000043 0 0.000000 2022-03-29T17:52:32.075 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Exiting state: Start, entered at 1648576351.2806172, 0.0 spent on 0 events 2022-03-29T17:52:32.075 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:32.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:32.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:32.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:32.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.076 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:32.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:11, acting: 0,1 2022-03-29T17:52:32.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:32.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:32.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:32.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 12:3 2022-03-29T17:52:32.078 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000184 0 0.000000 2022-03-29T17:52:32.078 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,280 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2807512, 0.0 spent on 0 events 2022-03-29T17:52:32.079 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:32.079 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:32.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:32.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:32.080 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:32.080 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:32.080 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:32.080 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.080 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:32.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:32.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:32.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:32.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:32.082 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000253 0 0.000000 2022-03-29T17:52:32.082 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2809956, 0.0 spent on 0 events 2022-03-29T17:52:32.082 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.082 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.082 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:32.083 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000047 0 0.000000 2022-03-29T17:52:32.083 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2812893, 0.0 spent on 0 events 2022-03-29T17:52:32.083 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.083 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:32.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.084 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean activate_map 2022-03-29T17:52:32.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: handle ActMap primary 2022-03-29T17:52:32.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean publish_stats_to_osd 18:12 2022-03-29T17:52:32.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean activate_map: Not dirtying info: last_persisted is 17 while current is 19 2022-03-29T17:52:32.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.087 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:32.087 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.018096 1 2022-03-29T17:52:31.264183+0000 2022-03-29T17:52:32.087 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2637331, 1648576351.2641835 spent on 1 events 2022-03-29T17:52:32.087 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:32.087 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Entering state: Started 2022-03-29T17:52:32.088 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:32.088 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Entering state: Start 2022-03-29T17:52:32.088 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:32.088 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:52:32.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,281 [shard 0] osd - Exiting state: Start, entered at 1648576351.2819014, 0.0 spent on 0 events 2022-03-29T17:52:32.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:32.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:32.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:32.089 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:32.090 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.090 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:32.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:14, acting: 0,1 2022-03-29T17:52:32.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:32.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:32.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:32.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 15:3 2022-03-29T17:52:32.092 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000187 0 0.000000 2022-03-29T17:52:32.092 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.2820446, 0.0 spent on 0 events 2022-03-29T17:52:32.092 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:32.092 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:32.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.093 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:32.093 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) 2022-03-29T17:52:32.093 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:32.094 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:32.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:32.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:32.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:32.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:32.095 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000240 0 0.000000 2022-03-29T17:52:32.095 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2822738, 0.0 spent on 0 events 2022-03-29T17:52:32.096 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.096 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:32.096 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000053 0 0.000000 2022-03-29T17:52:32.096 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2825644, 0.0 spent on 0 events 2022-03-29T17:52:32.097 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.097 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.098 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,282 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,282 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 18 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - peering_event(id=584, detail=PeeringEvent(from=1 pgid=3.15 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 18:12 2022-03-29T17:52:32.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 19 2022-03-29T17:52:32.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,283 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - peering_event(id=585, detail=PeeringEvent(from=1 pgid=3.10 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - peering_event(id=584, detail=PeeringEvent(from=1 pgid=3.15 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate_map 2022-03-29T17:52:32.103 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.012943 1 2022-03-29T17:52:31.271941+0000 2022-03-29T17:52:32.103 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Exiting state: Reset, entered at 1648576351.2714674, 1648576351.271942 spent on 1 events 2022-03-29T17:52:32.103 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started 2022-03-29T17:52:32.103 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Entering state: Started 2022-03-29T17:52:32.104 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Start 2022-03-29T17:52:32.104 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Entering state: Start 2022-03-29T17:52:32.104 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: transitioning to Primary 2022-03-29T17:52:32.104 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 2022-03-29T17:52:32.105 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Exiting state: Start, entered at 1648576351.284522, 0.0 spent on 0 events 2022-03-29T17:52:32.105 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary 2022-03-29T17:52:32.105 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Entering state: Started/Primary 2022-03-29T17:52:32.105 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering 2022-03-29T17:52:32.105 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Entering state: Started/Primary/Peering 2022-03-29T17:52:32.106 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.106 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo 2022-03-29T17:52:32.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1 2022-03-29T17:52:32.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:11, acting: 0,1 2022-03-29T17:52:32.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior prior osd.0 is down 2022-03-29T17:52:32.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 1 down 0 blocked_by {} 2022-03-29T17:52:32.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering up_thru 14 < same_since 19, must notify monitor 2022-03-29T17:52:32.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 18:15 2022-03-29T17:52:32.108 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetInfo 0.000187 0 0.000000 2022-03-29T17:52:32.108 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Exiting state: Started/Primary/Peering/GetInfo, entered at 1648576351.284666, 0.0 spent on 0 events 2022-03-29T17:52:32.108 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetLog 2022-03-29T17:52:32.108 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,284 [shard 0] osd - Entering state: Started/Primary/Peering/GetLog 2022-03-29T17:52:32.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,284 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting all_info osd.1 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:32.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering select_replicated_primary newest update on osd.1 with 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:32.109 INFO:tasks.ceph.osd.1.smithi138.stderr:up_primary: 1) selected as primary 2022-03-29T17:52:32.109 INFO:tasks.ceph.osd.1.smithi138.stderr:select_replicated_primary primary is osd.1 with 2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) 2022-03-29T17:52:32.109 INFO:tasks.ceph.osd.1.smithi138.stderr:calc_replicated_acting 2022-03-29T17:52:32.110 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:32.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated candidates by cost are: 2022-03-29T17:52:32.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_async_recovery_replicated result want={1} async_recovery= 2022-03-29T17:52:32.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering acting_recovery_backfill is 1 2022-03-29T17:52:32.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering choose_acting want={1} backfill_targets= async_recovery_targets= 2022-03-29T17:52:32.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: leaving GetLog 2022-03-29T17:52:32.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetLog 0.000245 0 0.000000 2022-03-29T17:52:32.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - Exiting state: Started/Primary/Peering/GetLog, entered at 1648576351.2848897, 0.0 spent on 0 events 2022-03-29T17:52:32.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.112 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - Entering state: Started/Primary/Peering/GetMissing 2022-03-29T17:52:32.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: still need up_thru update before going active 2022-03-29T17:52:32.112 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/GetMissing 0.000051 0 0.000000 2022-03-29T17:52:32.112 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - Exiting state: Started/Primary/Peering/GetMissing, entered at 1648576351.2851696, 0.0 spent on 0 events 2022-03-29T17:52:32.113 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.113 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - Entering state: Started/Primary/Peering/WaitUpThru 2022-03-29T17:52:32.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map: Not dirtying info: last_persisted is 12 while current is 19 2022-03-29T17:52:32.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.114 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,285 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - peering_event(id=586, detail=PeeringEvent(from=1 pgid=1.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - peering_event(id=587, detail=PeeringEvent(from=1 pgid=3.1c sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,285 [shard 0] osd - peering_event(id=585, detail=PeeringEvent(from=1 pgid=3.10 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=588, detail=PeeringEvent(from=1 pgid=2.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=586, detail=PeeringEvent(from=1 pgid=1.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=589, detail=PeeringEvent(from=1 pgid=2.6 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=590, detail=PeeringEvent(from=1 pgid=3.b sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=591, detail=PeeringEvent(from=1 pgid=3.12 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=587, detail=PeeringEvent(from=1 pgid=3.1c sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=589, detail=PeeringEvent(from=1 pgid=2.6 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=592, detail=PeeringEvent(from=1 pgid=3.f sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=588, detail=PeeringEvent(from=1 pgid=2.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - client_request(id=170, detail=m=[osd_op(client.4179.0:106 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e19) v8]): start 2022-03-29T17:52:32.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - client_request(id=170, detail=m=[osd_op(client.4179.0:106 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e19) v8]): in repeat 2022-03-29T17:52:32.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,286 [shard 0] osd - peering_event(id=592, detail=PeeringEvent(from=1 pgid=3.f sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - peering_event(id=590, detail=PeeringEvent(from=1 pgid=3.b sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - peering_event(id=593, detail=PeeringEvent(from=1 pgid=3.17 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - peering_event(id=591, detail=PeeringEvent(from=1 pgid=3.12 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - pg_advance_map(id=213, detail=PGAdvanceMap(pg=3.14 from=18 to=19)): complete 2022-03-29T17:52:32.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - peering_event(id=593, detail=PeeringEvent(from=1 pgid=3.17 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - peering_event(id=594, detail=PeeringEvent(from=1 pgid=3.a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - pg_advance_map(id=189, detail=PGAdvanceMap(pg=2.0 from=18 to=19)): complete 2022-03-29T17:52:32.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - peering_event(id=594, detail=PeeringEvent(from=1 pgid=3.a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - peering_event(id=595, detail=PeeringEvent(from=1 pgid=3.6 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - pg_advance_map(id=199, detail=PGAdvanceMap(pg=3.4 from=18 to=19)): complete 2022-03-29T17:52:32.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - peering_event(id=595, detail=PeeringEvent(from=1 pgid=3.6 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,287 [shard 0] osd - pg_advance_map(id=198, detail=PGAdvanceMap(pg=3.2 from=18 to=19)): complete 2022-03-29T17:52:32.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - peering_event(id=596, detail=PeeringEvent(from=1 pgid=3.c sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - peering_event(id=597, detail=PeeringEvent(from=1 pgid=3.13 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 2.6 2022-03-29T17:52:32.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - peering_event(id=589, detail=PeeringEvent(from=1 pgid=2.6 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - pg_advance_map(id=192, detail=PGAdvanceMap(pg=2.3 from=18 to=19)): complete 2022-03-29T17:52:32.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - peering_event(id=597, detail=PeeringEvent(from=1 pgid=3.13 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.1c 2022-03-29T17:52:32.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - peering_event(id=587, detail=PeeringEvent(from=1 pgid=3.1c sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.10 2022-03-29T17:52:32.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.124 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.124 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - peering_event(id=585, detail=PeeringEvent(from=1 pgid=3.10 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.124 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.124 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.15 2022-03-29T17:52:32.124 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - peering_event(id=584, detail=PeeringEvent(from=1 pgid=3.15 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - peering_event(id=596, detail=PeeringEvent(from=1 pgid=3.c sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.12 2022-03-29T17:52:32.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - peering_event(id=591, detail=PeeringEvent(from=1 pgid=3.12 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 1.0 2022-03-29T17:52:32.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - peering_event(id=586, detail=PeeringEvent(from=1 pgid=1.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.b 2022-03-29T17:52:32.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - peering_event(id=590, detail=PeeringEvent(from=1 pgid=3.b sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.a 2022-03-29T17:52:32.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - peering_event(id=594, detail=PeeringEvent(from=1 pgid=3.a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.17 2022-03-29T17:52:32.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - peering_event(id=593, detail=PeeringEvent(from=1 pgid=3.17 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.f 2022-03-29T17:52:32.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - peering_event(id=592, detail=PeeringEvent(from=1 pgid=3.f sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - pg_advance_map(id=190, detail=PGAdvanceMap(pg=2.1 from=18 to=19)): complete 2022-03-29T17:52:32.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - pg_advance_map(id=217, detail=PGAdvanceMap(pg=3.19 from=18 to=19)): complete 2022-03-29T17:52:32.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - pg_advance_map(id=202, detail=PGAdvanceMap(pg=3.7 from=18 to=19)): complete 2022-03-29T17:52:32.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - peering_event(id=598, detail=PeeringEvent(from=1 pgid=3.d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - peering_event(id=599, detail=PeeringEvent(from=1 pgid=2.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - peering_event(id=598, detail=PeeringEvent(from=1 pgid=3.d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.6 2022-03-29T17:52:32.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - peering_event(id=595, detail=PeeringEvent(from=1 pgid=3.6 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - peering_event(id=599, detail=PeeringEvent(from=1 pgid=2.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - peering_event(id=600, detail=PeeringEvent(from=1 pgid=2.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 2.7 2022-03-29T17:52:32.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - peering_event(id=588, detail=PeeringEvent(from=1 pgid=2.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - pg_advance_map(id=214, detail=PGAdvanceMap(pg=3.15 from=18 to=19)): complete 2022-03-29T17:52:32.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.13 2022-03-29T17:52:32.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,290 [shard 0] osd - peering_event(id=597, detail=PeeringEvent(from=1 pgid=3.13 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - peering_event(id=600, detail=PeeringEvent(from=1 pgid=2.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - peering_event(id=601, detail=PeeringEvent(from=1 pgid=3.1f sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_advance_map(id=215, detail=PGAdvanceMap(pg=3.16 from=18 to=19)): complete 2022-03-29T17:52:32.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - peering_event(id=601, detail=PeeringEvent(from=1 pgid=3.1f sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_advance_map(id=210, detail=PGAdvanceMap(pg=3.11 from=18 to=19)): complete 2022-03-29T17:52:32.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_advance_map(id=218, detail=PGAdvanceMap(pg=3.1a from=18 to=19)): complete 2022-03-29T17:52:32.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_advance_map(id=188, detail=PGAdvanceMap(pg=1.0 from=18 to=19)): complete 2022-03-29T17:52:32.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_advance_map(id=206, detail=PGAdvanceMap(pg=3.c from=18 to=19)): complete 2022-03-29T17:52:32.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_advance_map(id=221, detail=PGAdvanceMap(pg=3.1e from=18 to=19)): complete 2022-03-29T17:52:32.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_advance_map(id=200, detail=PGAdvanceMap(pg=3.5 from=18 to=19)): complete 2022-03-29T17:52:32.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_advance_map(id=220, detail=PGAdvanceMap(pg=3.1c from=18 to=19)): complete 2022-03-29T17:52:32.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.c 2022-03-29T17:52:32.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - peering_event(id=596, detail=PeeringEvent(from=1 pgid=3.c sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,291 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_advance_map(id=201, detail=PGAdvanceMap(pg=3.6 from=18 to=19)): complete 2022-03-29T17:52:32.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_advance_map(id=208, detail=PGAdvanceMap(pg=3.f from=18 to=19)): complete 2022-03-29T17:52:32.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 2.5 2022-03-29T17:52:32.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - peering_event(id=599, detail=PeeringEvent(from=1 pgid=2.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_advance_map(id=203, detail=PGAdvanceMap(pg=3.8 from=18 to=19)): complete 2022-03-29T17:52:32.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.d 2022-03-29T17:52:32.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - peering_event(id=598, detail=PeeringEvent(from=1 pgid=3.d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_advance_map(id=211, detail=PGAdvanceMap(pg=3.12 from=18 to=19)): complete 2022-03-29T17:52:32.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_advance_map(id=195, detail=PGAdvanceMap(pg=2.6 from=18 to=19)): complete 2022-03-29T17:52:32.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 2.4 2022-03-29T17:52:32.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - peering_event(id=600, detail=PeeringEvent(from=1 pgid=2.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_advance_map(id=209, detail=PGAdvanceMap(pg=3.10 from=18 to=19)): complete 2022-03-29T17:52:32.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.1f 2022-03-29T17:52:32.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - peering_event(id=601, detail=PeeringEvent(from=1 pgid=3.1f sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,292 [shard 0] osd - pg_advance_map(id=205, detail=PGAdvanceMap(pg=3.b from=18 to=19)): complete 2022-03-29T17:52:32.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - peering_event(id=602, detail=PeeringEvent(from=1 pgid=3.1b sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - peering_event(id=603, detail=PeeringEvent(from=1 pgid=2.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): start 2022-03-29T17:52:32.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - peering_event(id=602, detail=PeeringEvent(from=1 pgid=3.1b sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=204, detail=PGAdvanceMap(pg=3.a from=18 to=19)): complete 2022-03-29T17:52:32.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - peering_event(id=603, detail=PeeringEvent(from=1 pgid=2.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): pg present 2022-03-29T17:52:32.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=194, detail=PGAdvanceMap(pg=2.5 from=18 to=19)): complete 2022-03-29T17:52:32.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=216, detail=PGAdvanceMap(pg=3.17 from=18 to=19)): complete 2022-03-29T17:52:32.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=197, detail=PGAdvanceMap(pg=3.0 from=18 to=19)): complete 2022-03-29T17:52:32.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=222, detail=PGAdvanceMap(pg=3.1f from=18 to=19)): complete 2022-03-29T17:52:32.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=207, detail=PGAdvanceMap(pg=3.d from=18 to=19)): complete 2022-03-29T17:52:32.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=212, detail=PGAdvanceMap(pg=3.13 from=18 to=19)): complete 2022-03-29T17:52:32.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=196, detail=PGAdvanceMap(pg=2.7 from=18 to=19)): complete 2022-03-29T17:52:32.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=191, detail=PGAdvanceMap(pg=2.2 from=18 to=19)): complete 2022-03-29T17:52:32.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 3.1b 2022-03-29T17:52:32.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - peering_event(id=602, detail=PeeringEvent(from=1 pgid=3.1b sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - pg_advance_map(id=193, detail=PGAdvanceMap(pg=2.4 from=18 to=19)): complete 2022-03-29T17:52:32.151 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,293 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,293 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.151 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 IntervalFlush for pg: 2.2 2022-03-29T17:52:32.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Ending blocked outgoing recovery messages 2022-03-29T17:52:32.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - peering_event(id=603, detail=PeeringEvent(from=1 pgid=2.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): submitting ctx 2022-03-29T17:52:32.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - peering_event(id=589, detail=PeeringEvent(from=1 pgid=2.6 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - peering_event(id=587, detail=PeeringEvent(from=1 pgid=3.1c sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - pg_advance_map(id=219, detail=PGAdvanceMap(pg=3.1b from=18 to=19)): complete 2022-03-29T17:52:32.153 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - osd.1: now active 2022-03-29T17:52:32.153 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.154 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.154 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - peering_event(id=585, detail=PeeringEvent(from=1 pgid=3.10 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - peering_event(id=586, detail=PeeringEvent(from=1 pgid=1.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - peering_event(id=584, detail=PeeringEvent(from=1 pgid=3.15 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.155 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - client_request(id=168, detail=m=[osd_op(client.4179.0:90 3.a 3.fa1caca (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e19) v8]) same_interval_since: 19 2022-03-29T17:52:32.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - OpSequencer::start_op: op=168, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:32.156 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.157 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - peering_event(id=591, detail=PeeringEvent(from=1 pgid=3.12 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.157 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,294 [shard 0] osd - peering_event(id=593, detail=PeeringEvent(from=1 pgid=3.17 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=594, detail=PeeringEvent(from=1 pgid=3.a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=590, detail=PeeringEvent(from=1 pgid=3.b sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.158 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - client_request(id=169, detail=m=[osd_op(client.4179.0:105 3.15 3.f1c01795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e19) v8]) same_interval_since: 19 2022-03-29T17:52:32.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - OpSequencer::start_op: op=169, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:32.159 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.160 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=597, detail=PeeringEvent(from=1 pgid=3.13 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=592, detail=PeeringEvent(from=1 pgid=3.f sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=588, detail=PeeringEvent(from=1 pgid=2.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.161 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.161 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.161 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.161 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=599, detail=PeeringEvent(from=1 pgid=2.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.162 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=598, detail=PeeringEvent(from=1 pgid=3.d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - client_request(id=170, detail=m=[osd_op(client.4179.0:106 3.a 3.af2782ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e19) v8]) same_interval_since: 19 2022-03-29T17:52:32.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - OpSequencer::start_op: op=170, last_started=168, last_unblocked=168, last_completed=0 2022-03-29T17:52:32.163 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.164 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.164 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=601, detail=PeeringEvent(from=1 pgid=3.1f sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=602, detail=PeeringEvent(from=1 pgid=3.1b sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=596, detail=PeeringEvent(from=1 pgid=3.c sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - peering_event(id=595, detail=PeeringEvent(from=1 pgid=3.6 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.165 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,295 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,296 [shard 0] osd - peering_event(id=603, detail=PeeringEvent(from=1 pgid=2.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,296 [shard 0] osd - peering_event(id=600, detail=PeeringEvent(from=1 pgid=2.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 IntervalFlush)): complete 2022-03-29T17:52:32.168 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,470 [shard 0] osd - peering_event(id=307, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.168 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,470 [shard 0] osd - peering_event(id=307, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.168 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,470 [shard 0] osd - do_peering_event ignoring epoch_sent: 14 epoch_requested: 14 RenewLease -- pg has reset 2022-03-29T17:52:32.168 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,470 [shard 0] osd - peering_event(id=307, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.169 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,470 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.169 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,471 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.169 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,471 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.169 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,471 [shard 0] osd - peering_event(id=307, detail=PeeringEvent(from=2 pgid=3.1d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.170 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,490 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.170 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,490 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.170 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,490 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:52:32.170 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,490 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.117797852s 2022-03-29T17:52:32.170 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,490 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.117797852s (sent 174.123367310s) 2022-03-29T17:52:32.171 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,490 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.171 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,490 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.171 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,490 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,491 [shard 0] osd - handle_peering_op on 3.8 from 2 2022-03-29T17:52:32.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,491 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.117797852s ub 174.123367310s int 16.000000000s))): start 2022-03-29T17:52:32.172 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,491 [shard 0] osd - peering_event(id=308, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,491 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.117797852s ub 174.123367310s int 16.000000000s))): got map 19 2022-03-29T17:52:32.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,491 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.117797852s ub 174.123367310s int 16.000000000s))): pg present 2022-03-29T17:52:32.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,492 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.117797852s ub 174.123367310s int 16.000000000s) for pg: 3.8 2022-03-29T17:52:32.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,492 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.117797852s ub 174.123367310s int 16.000000000s) 2022-03-29T17:52:32.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,492 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.188064575s 2022-03-29T17:52:32.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,492 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.188064575s 2022-03-29T17:52:32.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,492 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,492 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.117797852s ub 174.123367310s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,492 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,492 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.117797852s ub 174.123367310s int 16.000000000s))): complete 2022-03-29T17:52:32.179 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - peering_event(id=309, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.179 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.179 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - peering_event(id=309, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.180 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.180 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - do_peering_event ignoring epoch_sent: 14 epoch_requested: 14 RenewLease -- pg has reset 2022-03-29T17:52:32.180 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - peering_event(id=309, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.180 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.181 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:52:32.181 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 166.128143311s 2022-03-29T17:52:32.181 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 166.128143311s (sent 174.133697510s) 2022-03-29T17:52:32.181 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.181 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.182 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,500 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.182 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,501 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.182 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.182 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - peering_event(id=309, detail=PeeringEvent(from=2 pgid=3.18 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.182 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - handle_peering_op on 3.8 from 1 2022-03-29T17:52:32.183 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=1 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.123367310s))): start 2022-03-29T17:52:32.183 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - peering_event(id=310, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.183 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=1 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.123367310s))): got map 19 2022-03-29T17:52:32.183 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=1 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.123367310s))): pg present 2022-03-29T17:52:32.184 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.123367310s) for pg: 3.8 2022-03-29T17:52:32.184 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:52:32.184 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 174.123367310s) 2022-03-29T17:52:32.184 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.184 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=1 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.123367310s))): submitting ctx 2022-03-29T17:52:32.185 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,501 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,502 [shard 0] osd - handle_peering_op on 3.7 from 2 2022-03-29T17:52:32.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,502 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.128143311s ub 174.133697510s int 16.000000000s))): start 2022-03-29T17:52:32.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,502 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.128143311s ub 174.133697510s int 16.000000000s))): got map 19 2022-03-29T17:52:32.186 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,502 [shard 0] osd - peering_event(id=311, detail=PeeringEvent(from=1 pgid=3.8 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.123367310s))): complete 2022-03-29T17:52:32.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,503 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.128143311s ub 174.133697510s int 16.000000000s))): pg present 2022-03-29T17:52:32.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,503 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.128143311s ub 174.133697510s int 16.000000000s) for pg: 3.7 2022-03-29T17:52:32.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,503 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active proc_lease pg_lease(ru 166.128143311s ub 174.133697510s int 16.000000000s) 2022-03-29T17:52:32.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,503 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active no peer_clock_delta_lb -> ruub 174.199264526s 2022-03-29T17:52:32.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,503 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active proc_lease readable_until_ub now 174.199264526s 2022-03-29T17:52:32.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,503 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,503 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.128143311s ub 174.133697510s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,503 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.189 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,504 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.128143311s ub 174.133697510s int 16.000000000s))): complete 2022-03-29T17:52:32.192 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,530 [shard 0] osd - peering_event(id=312, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.192 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,530 [shard 0] osd - peering_event(id=312, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.192 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,530 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:52:32.192 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,530 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.158020020s 2022-03-29T17:52:32.193 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,530 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.158020020s (sent 174.163772583s) 2022-03-29T17:52:32.193 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,530 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.193 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,530 [shard 0] osd - peering_event(id=312, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.193 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,530 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.194 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - handle_peering_op on 3.7 from 1 2022-03-29T17:52:32.194 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=1 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.133697510s))): start 2022-03-29T17:52:32.194 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=1 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.133697510s))): got map 19 2022-03-29T17:52:32.194 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=1 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.133697510s))): pg present 2022-03-29T17:52:32.194 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - peering_event(id=312, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.195 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.133697510s) for pg: 3.7 2022-03-29T17:52:32.195 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:52:32.195 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 174.133697510s) 2022-03-29T17:52:32.195 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.196 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=1 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.133697510s))): submitting ctx 2022-03-29T17:52:32.196 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,531 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,532 [shard 0] osd - handle_peering_op on 3.1e from 2 2022-03-29T17:52:32.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,532 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.158020020s ub 174.163772583s int 16.000000000s))): start 2022-03-29T17:52:32.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,532 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.158020020s ub 174.163772583s int 16.000000000s))): got map 19 2022-03-29T17:52:32.197 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,532 [shard 0] osd - peering_event(id=313, detail=PeeringEvent(from=1 pgid=3.7 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.133697510s))): complete 2022-03-29T17:52:32.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.158020020s ub 174.163772583s int 16.000000000s))): pg present 2022-03-29T17:52:32.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.158020020s ub 174.163772583s int 16.000000000s) for pg: 3.1e 2022-03-29T17:52:32.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.158020020s ub 174.163772583s int 16.000000000s) 2022-03-29T17:52:32.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.229187012s 2022-03-29T17:52:32.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.229187012s 2022-03-29T17:52:32.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.158020020s ub 174.163772583s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,533 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.158020020s ub 174.163772583s int 16.000000000s))): complete 2022-03-29T17:52:32.203 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,550 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.203 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,550 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,550 [shard 0] osd - do_peering_event ignoring epoch_sent: 14 epoch_requested: 14 RenewLease -- pg has reset 2022-03-29T17:52:32.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,550 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.204 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,550 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.204 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,551 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,551 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,551 [shard 0] osd - peering_event(id=314, detail=PeeringEvent(from=2 pgid=3.e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,560 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,560 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.205 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,560 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:52:32.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,560 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.188369751s 2022-03-29T17:52:32.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,560 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.188369751s (sent 174.193771362s) 2022-03-29T17:52:32.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,560 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.206 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,560 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.211 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,560 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.211 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - handle_peering_op on 3.1e from 1 2022-03-29T17:52:32.211 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=1 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.163772583s))): start 2022-03-29T17:52:32.212 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - peering_event(id=315, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.212 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=1 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.163772583s))): got map 19 2022-03-29T17:52:32.212 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=1 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.163772583s))): pg present 2022-03-29T17:52:32.212 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.163772583s) for pg: 3.1e 2022-03-29T17:52:32.212 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:52:32.213 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 174.163772583s) 2022-03-29T17:52:32.213 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.213 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=1 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.163772583s))): submitting ctx 2022-03-29T17:52:32.213 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,561 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,562 [shard 0] osd - handle_peering_op on 3.16 from 2 2022-03-29T17:52:32.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,562 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.188369751s ub 174.193771362s int 16.000000000s))): start 2022-03-29T17:52:32.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,562 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.188369751s ub 174.193771362s int 16.000000000s))): got map 19 2022-03-29T17:52:32.215 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,562 [shard 0] osd - peering_event(id=316, detail=PeeringEvent(from=1 pgid=3.1e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.163772583s))): complete 2022-03-29T17:52:32.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.188369751s ub 174.193771362s int 16.000000000s))): pg present 2022-03-29T17:52:32.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.188369751s ub 174.193771362s int 16.000000000s) for pg: 3.16 2022-03-29T17:52:32.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.188369751s ub 174.193771362s int 16.000000000s) 2022-03-29T17:52:32.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.259246826s 2022-03-29T17:52:32.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.259246826s 2022-03-29T17:52:32.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.188369751s ub 174.193771362s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - peering_event(id=608, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - peering_event(id=608, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.19 2022-03-29T17:52:32.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.256790161s 2022-03-29T17:52:32.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.256790161s (sent 174.259719849s) 2022-03-29T17:52:32.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - peering_event(id=608, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,563 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,564 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.188369751s ub 174.193771362s int 16.000000000s))): complete 2022-03-29T17:52:32.221 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,564 [shard 0] osd - peering_event(id=608, detail=PeeringEvent(from=1 pgid=3.19 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.224 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,570 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,570 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,570 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:52:32.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,570 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.198211670s 2022-03-29T17:52:32.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,570 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.198211670s (sent 174.203826904s) 2022-03-29T17:52:32.225 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,570 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.226 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,570 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.226 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,570 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.226 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - handle_peering_op on 3.16 from 1 2022-03-29T17:52:32.226 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=1 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.193771362s))): start 2022-03-29T17:52:32.227 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=1 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.193771362s))): got map 19 2022-03-29T17:52:32.227 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - peering_event(id=317, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.227 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=1 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.193771362s))): pg present 2022-03-29T17:52:32.227 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.193771362s) for pg: 3.16 2022-03-29T17:52:32.227 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:52:32.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 174.193771362s) 2022-03-29T17:52:32.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=1 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.193771362s))): submitting ctx 2022-03-29T17:52:32.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,571 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.228 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - handle_peering_op on 3.19 from 1 2022-03-29T17:52:32.229 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=1 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.256790161s ub 174.259719849s int 16.000000000s))): start 2022-03-29T17:52:32.229 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=1 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.256790161s ub 174.259719849s int 16.000000000s))): got map 19 2022-03-29T17:52:32.229 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=1 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.256790161s ub 174.259719849s int 16.000000000s))): pg present 2022-03-29T17:52:32.229 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - peering_event(id=318, detail=PeeringEvent(from=1 pgid=3.16 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.193771362s))): complete 2022-03-29T17:52:32.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.256790161s ub 174.259719849s int 16.000000000s) for pg: 3.19 2022-03-29T17:52:32.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.256790161s ub 174.259719849s int 16.000000000s) 2022-03-29T17:52:32.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.205581665s 2022-03-29T17:52:32.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.205581665s 2022-03-29T17:52:32.230 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.231 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=1 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.256790161s ub 174.259719849s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.231 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,572 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,573 [shard 0] osd - handle_peering_op on 3.5 from 2 2022-03-29T17:52:32.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,573 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.198211670s ub 174.203826904s int 16.000000000s))): start 2022-03-29T17:52:32.232 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,573 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.198211670s ub 174.203826904s int 16.000000000s))): got map 19 2022-03-29T17:52:32.232 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,573 [shard 0] osd - peering_event(id=319, detail=PeeringEvent(from=1 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.256790161s ub 174.259719849s int 16.000000000s))): complete 2022-03-29T17:52:32.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.198211670s ub 174.203826904s int 16.000000000s))): pg present 2022-03-29T17:52:32.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.198211670s ub 174.203826904s int 16.000000000s) for pg: 3.5 2022-03-29T17:52:32.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.198211670s ub 174.203826904s int 16.000000000s) 2022-03-29T17:52:32.234 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.270202637s 2022-03-29T17:52:32.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.270202637s 2022-03-29T17:52:32.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.198211670s ub 174.203826904s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.235 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.237 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - handle_peering_op on 3.5 from 1 2022-03-29T17:52:32.237 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=1 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.203826904s))): start 2022-03-29T17:52:32.237 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=1 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.203826904s))): got map 19 2022-03-29T17:52:32.237 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,574 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=1 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.203826904s))): pg present 2022-03-29T17:52:32.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.203826904s) for pg: 3.5 2022-03-29T17:52:32.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:52:32.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 174.203826904s) 2022-03-29T17:52:32.238 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.239 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=1 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.203826904s))): submitting ctx 2022-03-29T17:52:32.239 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.239 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.198211670s ub 174.203826904s int 16.000000000s))): complete 2022-03-29T17:52:32.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - handle_peering_op on 3.19 from 2 2022-03-29T17:52:32.240 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,575 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.259719849s))): start 2022-03-29T17:52:32.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - peering_event(id=320, detail=PeeringEvent(from=1 pgid=3.5 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.203826904s))): complete 2022-03-29T17:52:32.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.259719849s))): got map 19 2022-03-29T17:52:32.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.259719849s))): pg present 2022-03-29T17:52:32.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.259719849s) for pg: 3.19 2022-03-29T17:52:32.242 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 174.259719849s 2022-03-29T17:52:32.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.259719849s (sent 174.259719849s) 2022-03-29T17:52:32.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.259719849s))): submitting ctx 2022-03-29T17:52:32.243 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,576 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.244 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,577 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.19 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.259719849s))): complete 2022-03-29T17:52:32.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,600 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,600 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,601 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:52:32.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,601 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 166.228576660s 2022-03-29T17:52:32.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,601 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.228576660s (sent 174.234085083s) 2022-03-29T17:52:32.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,601 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,601 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,601 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,601 [shard 0] osd - handle_peering_op on 3.11 from 2 2022-03-29T17:52:32.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,601 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.228576660s ub 174.234085083s int 16.000000000s))): start 2022-03-29T17:52:32.247 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.228576660s ub 174.234085083s int 16.000000000s))): got map 19 2022-03-29T17:52:32.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - peering_event(id=321, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.251 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.228576660s ub 174.234085083s int 16.000000000s))): pg present 2022-03-29T17:52:32.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.228576660s ub 174.234085083s int 16.000000000s) for pg: 3.11 2022-03-29T17:52:32.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.228576660s ub 174.234085083s int 16.000000000s) 2022-03-29T17:52:32.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.298614502s 2022-03-29T17:52:32.252 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.298614502s 2022-03-29T17:52:32.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.228576660s ub 174.234085083s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,602 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,603 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.2 pg_lease(ru 166.228576660s ub 174.234085083s int 16.000000000s))): complete 2022-03-29T17:52:32.253 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,614 [shard 0] osd - peering_event(id=612, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,614 [shard 0] osd - peering_event(id=612, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,614 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.14 2022-03-29T17:52:32.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,614 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.308303833s 2022-03-29T17:52:32.254 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,614 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.308303833s (sent 174.310256958s) 2022-03-29T17:52:32.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,614 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,614 [shard 0] osd - peering_event(id=612, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,614 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,615 [shard 0] osd - peering_event(id=612, detail=PeeringEvent(from=1 pgid=3.14 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.255 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - peering_event(id=613, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - peering_event(id=614, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - peering_event(id=614, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - peering_event(id=613, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.256 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.0 2022-03-29T17:52:32.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.328613281s 2022-03-29T17:52:32.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.328613281s (sent 174.330596924s) 2022-03-29T17:52:32.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - peering_event(id=613, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.257 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,634 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,635 [shard 0] osd - do_peering_event ignoring epoch_sent: 14 epoch_requested: 14 RenewLease -- pg has reset 2022-03-29T17:52:32.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,635 [shard 0] osd - peering_event(id=614, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,635 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.258 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,635 [shard 0] osd - peering_event(id=613, detail=PeeringEvent(from=1 pgid=3.0 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.258 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,638 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,638 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.259 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,638 [shard 0] osd - peering_event(id=614, detail=PeeringEvent(from=1 pgid=3.1c sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,642 [shard 0] osd - handle_peering_op on 3.11 from 1 2022-03-29T17:52:32.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,642 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=1 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.234085083s))): start 2022-03-29T17:52:32.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,642 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=1 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.234085083s))): got map 19 2022-03-29T17:52:32.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,642 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=1 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.234085083s))): pg present 2022-03-29T17:52:32.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - handle_peering_op on 3.14 from 1 2022-03-29T17:52:32.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=1 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.308303833s ub 174.310256958s int 16.000000000s))): start 2022-03-29T17:52:32.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.234085083s) for pg: 3.11 2022-03-29T17:52:32.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:52:32.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 174.234085083s) 2022-03-29T17:52:32.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=1 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.234085083s))): submitting ctx 2022-03-29T17:52:32.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=1 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.308303833s ub 174.310256958s int 16.000000000s))): got map 19 2022-03-29T17:52:32.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=1 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.308303833s ub 174.310256958s int 16.000000000s))): pg present 2022-03-29T17:52:32.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - handle_peering_op on 3.0 from 1 2022-03-29T17:52:32.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,643 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=1 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.328613281s ub 174.330596924s int 16.000000000s))): start 2022-03-29T17:52:32.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.308303833s ub 174.310256958s int 16.000000000s) for pg: 3.14 2022-03-29T17:52:32.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.308303833s ub 174.310256958s int 16.000000000s) 2022-03-29T17:52:32.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.277175903s 2022-03-29T17:52:32.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.277175903s 2022-03-29T17:52:32.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=1 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.308303833s ub 174.310256958s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - peering_event(id=322, detail=PeeringEvent(from=1 pgid=3.11 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.1 pg_lease_ack(ruub 174.234085083s))): complete 2022-03-29T17:52:32.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=1 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.328613281s ub 174.330596924s int 16.000000000s))): got map 19 2022-03-29T17:52:32.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=1 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.328613281s ub 174.330596924s int 16.000000000s))): pg present 2022-03-29T17:52:32.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.328613281s ub 174.330596924s int 16.000000000s) for pg: 3.0 2022-03-29T17:52:32.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.328613281s ub 174.330596924s int 16.000000000s) 2022-03-29T17:52:32.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.277755737s 2022-03-29T17:52:32.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.277755737s 2022-03-29T17:52:32.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=1 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.328613281s ub 174.330596924s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,644 [shard 0] osd - peering_event(id=323, detail=PeeringEvent(from=1 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.308303833s ub 174.310256958s int 16.000000000s))): complete 2022-03-29T17:52:32.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,645 [shard 0] osd - peering_event(id=324, detail=PeeringEvent(from=1 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.328613281s ub 174.330596924s int 16.000000000s))): complete 2022-03-29T17:52:32.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - peering_event(id=615, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - peering_event(id=616, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.272 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - peering_event(id=615, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - peering_event(id=616, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1a 2022-03-29T17:52:32.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until peer osd.2 ruub 166.348846436s 2022-03-29T17:52:32.273 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until readable_until[_ub] 166.348846436s (sent 174.351730347s) 2022-03-29T17:52:32.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - peering_event(id=616, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,655 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - do_peering_event ignoring epoch_sent: 14 epoch_requested: 14 RenewLease -- pg has reset 2022-03-29T17:52:32.274 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - peering_event(id=615, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.275 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - handle_peering_op on 3.1a from 1 2022-03-29T17:52:32.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=1 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.348846436s ub 174.351730347s int 16.000000000s))): start 2022-03-29T17:52:32.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=1 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.348846436s ub 174.351730347s int 16.000000000s))): got map 19 2022-03-29T17:52:32.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=1 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.348846436s ub 174.351730347s int 16.000000000s))): pg present 2022-03-29T17:52:32.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.348846436s ub 174.351730347s int 16.000000000s) for pg: 3.1a 2022-03-29T17:52:32.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active proc_lease pg_lease(ru 166.348846436s ub 174.351730347s int 16.000000000s) 2022-03-29T17:52:32.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,656 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active no peer_clock_delta_lb -> ruub 174.289993286s 2022-03-29T17:52:32.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active proc_lease readable_until_ub now 174.289993286s 2022-03-29T17:52:32.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=1 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.348846436s ub 174.351730347s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.279 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,657 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - peering_event(id=615, detail=PeeringEvent(from=1 pgid=3.b sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - peering_event(id=616, detail=PeeringEvent(from=1 pgid=3.1a sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.280 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - handle_peering_op on 3.14 from 2 2022-03-29T17:52:32.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,657 [shard 0] osd - peering_event(id=325, detail=PeeringEvent(from=1 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.348846436s ub 174.351730347s int 16.000000000s))): complete 2022-03-29T17:52:32.287 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,658 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.310256958s))): start 2022-03-29T17:52:32.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,658 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.310256958s))): got map 19 2022-03-29T17:52:32.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,658 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.310256958s))): pg present 2022-03-29T17:52:32.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,658 [shard 0] osd - handle_peering_op on 3.0 from 2 2022-03-29T17:52:32.288 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,658 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.330596924s))): start 2022-03-29T17:52:32.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,659 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.310256958s) for pg: 3.14 2022-03-29T17:52:32.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,659 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 174.310256958s 2022-03-29T17:52:32.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,659 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.310256958s (sent 174.310256958s) 2022-03-29T17:52:32.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,659 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.289 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,659 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.310256958s))): submitting ctx 2022-03-29T17:52:32.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,659 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,659 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.330596924s))): got map 19 2022-03-29T17:52:32.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,659 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.330596924s))): pg present 2022-03-29T17:52:32.290 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.330596924s) for pg: 3.0 2022-03-29T17:52:32.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 174.330596924s 2022-03-29T17:52:32.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.330596924s (sent 174.330596924s) 2022-03-29T17:52:32.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.330596924s))): submitting ctx 2022-03-29T17:52:32.291 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.14 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.310256958s))): complete 2022-03-29T17:52:32.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - handle_peering_op on 3.1a from 2 2022-03-29T17:52:32.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.351730347s))): start 2022-03-29T17:52:32.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.351730347s))): got map 19 2022-03-29T17:52:32.292 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.351730347s))): pg present 2022-03-29T17:52:32.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.0 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.330596924s))): complete 2022-03-29T17:52:32.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.351730347s) for pg: 3.1a 2022-03-29T17:52:32.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,660 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until peer osd.2 ruub 174.351730347s 2022-03-29T17:52:32.293 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,661 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean recalc_readable_until readable_until[_ub] 174.351730347s (sent 174.351730347s) 2022-03-29T17:52:32.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,661 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,661 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.351730347s))): submitting ctx 2022-03-29T17:52:32.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,661 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,661 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.1a sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.351730347s))): complete 2022-03-29T17:52:32.294 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - peering_event(id=620, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - peering_event(id=621, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - peering_event(id=621, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - peering_event(id=620, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.295 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - do_peering_event ignoring epoch_sent: 14 epoch_requested: 14 RenewLease -- pg has reset 2022-03-29T17:52:32.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - peering_event(id=621, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.2 2022-03-29T17:52:32.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.358886719s 2022-03-29T17:52:32.296 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.358886719s (sent 174.361740112s) 2022-03-29T17:52:32.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,665 [shard 0] osd - peering_event(id=620, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,666 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.297 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,666 [shard 0] osd - peering_event(id=620, detail=PeeringEvent(from=1 pgid=3.2 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.297 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,666 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,666 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,666 [shard 0] osd - peering_event(id=621, detail=PeeringEvent(from=1 pgid=3.13 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - peering_event(id=622, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.298 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - peering_event(id=623, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - peering_event(id=623, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - peering_event(id=622, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - do_peering_event ignoring epoch_sent: 14 epoch_requested: 14 RenewLease -- pg has reset 2022-03-29T17:52:32.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - peering_event(id=623, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.299 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.4 2022-03-29T17:52:32.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 166.388748169s 2022-03-29T17:52:32.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 166.388748169s (sent 174.391754150s) 2022-03-29T17:52:32.300 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - peering_event(id=622, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,695 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.301 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,696 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,696 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.301 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,696 [shard 0] osd - peering_event(id=623, detail=PeeringEvent(from=1 pgid=3.d sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.302 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,697 [shard 0] osd - peering_event(id=622, detail=PeeringEvent(from=1 pgid=3.4 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,699 [shard 0] osd - handle_peering_op on 3.2 from 1 2022-03-29T17:52:32.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,699 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=1 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.358886719s ub 174.361740112s int 16.000000000s))): start 2022-03-29T17:52:32.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,699 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=1 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.358886719s ub 174.361740112s int 16.000000000s))): got map 19 2022-03-29T17:52:32.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,699 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=1 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.358886719s ub 174.361740112s int 16.000000000s))): pg present 2022-03-29T17:52:32.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,699 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.358886719s ub 174.361740112s int 16.000000000s) for pg: 3.2 2022-03-29T17:52:32.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,699 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.358886719s ub 174.361740112s int 16.000000000s) 2022-03-29T17:52:32.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.333053589s 2022-03-29T17:52:32.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.333053589s 2022-03-29T17:52:32.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=1 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.358886719s ub 174.361740112s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - handle_peering_op on 3.4 from 1 2022-03-29T17:52:32.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.388748169s ub 174.391754150s int 16.000000000s))): start 2022-03-29T17:52:32.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.388748169s ub 174.391754150s int 16.000000000s))): got map 19 2022-03-29T17:52:32.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.388748169s ub 174.391754150s int 16.000000000s))): pg present 2022-03-29T17:52:32.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - peering_event(id=326, detail=PeeringEvent(from=1 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.358886719s ub 174.361740112s int 16.000000000s))): complete 2022-03-29T17:52:32.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.388748169s ub 174.391754150s int 16.000000000s) for pg: 3.4 2022-03-29T17:52:32.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease pg_lease(ru 166.388748169s ub 174.391754150s int 16.000000000s) 2022-03-29T17:52:32.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active no peer_clock_delta_lb -> ruub 174.333892822s 2022-03-29T17:52:32.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active proc_lease readable_until_ub now 174.333892822s 2022-03-29T17:52:32.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.388748169s ub 174.391754150s int 16.000000000s))): submitting ctx 2022-03-29T17:52:32.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,700 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:31,701 [shard 0] osd - peering_event(id=327, detail=PeeringEvent(from=1 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLease epoch 19 from osd.1 pg_lease(ru 166.388748169s ub 174.391754150s int 16.000000000s))): complete 2022-03-29T17:52:32.313 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,715 [shard 0] osd - peering_event(id=624, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:32.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,715 [shard 0] osd - peering_event(id=624, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:32.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,715 [shard 0] osd - do_peering_event ignoring epoch_sent: 14 epoch_requested: 14 RenewLease -- pg has reset 2022-03-29T17:52:32.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,715 [shard 0] osd - peering_event(id=624, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:32.314 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,715 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.315 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,716 [shard 0] osd - send_alive want=19 up_thru_wanted=19 2022-03-29T17:52:32.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,716 [shard 0] osd - send_alive want=19 <= up_thru_wanted=19; skipping 2022-03-29T17:52:32.315 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,716 [shard 0] osd - peering_event(id=624, detail=PeeringEvent(from=1 pgid=3.10 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:32.315 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,720 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:32.315 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,721 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:32.316 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,721 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:32.316 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:31,721 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:32.316 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:31,721 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:32.316 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:31,721 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:32.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,740 [shard 0] osd - handle_peering_op on 3.2 from 2 2022-03-29T17:52:32.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,740 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.361740112s))): start 2022-03-29T17:52:32.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,740 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.361740112s))): got map 19 2022-03-29T17:52:32.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,740 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.361740112s))): pg present 2022-03-29T17:52:32.317 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.361740112s) for pg: 3.2 2022-03-29T17:52:32.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 174.361740112s 2022-03-29T17:52:32.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.361740112s (sent 174.361740112s) 2022-03-29T17:52:32.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.361740112s))): submitting ctx 2022-03-29T17:52:32.318 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - handle_peering_op on 3.4 from 2 2022-03-29T17:52:32.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.391754150s))): start 2022-03-29T17:52:32.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.391754150s))): got map 19 2022-03-29T17:52:32.319 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.391754150s))): pg present 2022-03-29T17:52:32.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.391754150s) for pg: 3.4 2022-03-29T17:52:32.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,741 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.2 ruub 174.391754150s 2022-03-29T17:52:32.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,742 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.391754150s (sent 174.391754150s) 2022-03-29T17:52:32.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,742 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.320 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,742 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.391754150s))): submitting ctx 2022-03-29T17:52:32.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,742 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,742 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.2 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.361740112s))): complete 2022-03-29T17:52:32.321 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:31,742 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.4 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 MLeaseAck epoch 19 from osd.2 pg_lease_ack(ruub 174.391754150s))): complete 2022-03-29T17:52:32.322 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,753 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:32.322 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,754 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:32.323 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,754 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:32.323 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:31,754 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] waiting 1.6 seconds ... 2022-03-29T17:52:32.323 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:31,754 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:32.323 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:31,754 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] waiting 1.6 seconds ... 2022-03-29T17:52:32.324 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,028 [shard 0] osd - Heartbeat::Peer: osd.0 removed 2022-03-29T17:52:32.324 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,028 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:32.325 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,028 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:32.325 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,028 [shard 0] alienstore - stat 2022-03-29T17:52:32.325 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,028 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.138:6808/35578 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:32.325 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,028 [shard 0] ms - [osd.1(hb_back) v2:172.21.15.138:6807/35578 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:32.326 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,034 [shard 0] osd - Heartbeat::Peer: osd.0 removed 2022-03-29T17:52:32.326 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] closing: reset no, replace no 2022-03-29T17:52:32.327 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,034 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] closing: reset no, replace no 2022-03-29T17:52:32.327 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,034 [shard 0] alienstore - stat 2022-03-29T17:52:32.327 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,034 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.0 v2:172.21.15.138:6809/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:32.327 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.0 v2:172.21.15.138:6806/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:32.328 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,211 [shard 0] osd - handle_osd_map osd_map(19..20 src has 1..20) v4 2022-03-29T17:52:32.329 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,211 [shard 0] osd - handle_osd_map epochs [19..20], i have 19, src has [1..20] 2022-03-29T17:52:32.329 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,212 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:32.329 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,213 [shard 0] osd - osd.1: committed_osd_maps(20, 20) 2022-03-29T17:52:32.329 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,213 [shard 0] osd - osd.1: map e 20 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:32.330 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,214 [shard 0] osd - handle_osd_map osd_map(20..20 src has 1..20) v4 2022-03-29T17:52:32.330 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,214 [shard 0] osd - handle_osd_map epochs [20..20], i have 20, src has [1..20] 2022-03-29T17:52:32.330 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,214 [shard 0] osd - handle_osd_map osd_map(20..20 src has 1..20) v4 2022-03-29T17:52:32.331 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,214 [shard 0] osd - handle_osd_map epochs [20..20], i have 19, src has [1..20] 2022-03-29T17:52:32.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - OSD::handle_osd_map: do_transaction... 2022-03-29T17:52:32.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=223, detail=PGAdvanceMap(pg=1.0 from=19 to=20)): start 2022-03-29T17:52:32.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=224, detail=PGAdvanceMap(pg=2.0 from=19 to=20)): start 2022-03-29T17:52:32.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=225, detail=PGAdvanceMap(pg=2.1 from=19 to=20)): start 2022-03-29T17:52:32.339 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=226, detail=PGAdvanceMap(pg=2.2 from=19 to=20)): start 2022-03-29T17:52:32.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=227, detail=PGAdvanceMap(pg=2.3 from=19 to=20)): start 2022-03-29T17:52:32.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=228, detail=PGAdvanceMap(pg=2.4 from=19 to=20)): start 2022-03-29T17:52:32.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=229, detail=PGAdvanceMap(pg=2.5 from=19 to=20)): start 2022-03-29T17:52:32.340 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=230, detail=PGAdvanceMap(pg=2.6 from=19 to=20)): start 2022-03-29T17:52:32.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=231, detail=PGAdvanceMap(pg=2.7 from=19 to=20)): start 2022-03-29T17:52:32.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=232, detail=PGAdvanceMap(pg=3.0 from=19 to=20)): start 2022-03-29T17:52:32.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=233, detail=PGAdvanceMap(pg=3.2 from=19 to=20)): start 2022-03-29T17:52:32.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=234, detail=PGAdvanceMap(pg=3.4 from=19 to=20)): start 2022-03-29T17:52:32.341 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=235, detail=PGAdvanceMap(pg=3.5 from=19 to=20)): start 2022-03-29T17:52:32.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=236, detail=PGAdvanceMap(pg=3.6 from=19 to=20)): start 2022-03-29T17:52:32.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=237, detail=PGAdvanceMap(pg=3.7 from=19 to=20)): start 2022-03-29T17:52:32.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,215 [shard 0] osd - pg_advance_map(id=238, detail=PGAdvanceMap(pg=3.8 from=19 to=20)): start 2022-03-29T17:52:32.342 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=239, detail=PGAdvanceMap(pg=3.a from=19 to=20)): start 2022-03-29T17:52:32.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=240, detail=PGAdvanceMap(pg=3.b from=19 to=20)): start 2022-03-29T17:52:32.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=241, detail=PGAdvanceMap(pg=3.c from=19 to=20)): start 2022-03-29T17:52:32.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=242, detail=PGAdvanceMap(pg=3.d from=19 to=20)): start 2022-03-29T17:52:32.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=243, detail=PGAdvanceMap(pg=3.f from=19 to=20)): start 2022-03-29T17:52:32.343 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=244, detail=PGAdvanceMap(pg=3.10 from=19 to=20)): start 2022-03-29T17:52:32.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=245, detail=PGAdvanceMap(pg=3.11 from=19 to=20)): start 2022-03-29T17:52:32.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=246, detail=PGAdvanceMap(pg=3.12 from=19 to=20)): start 2022-03-29T17:52:32.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=247, detail=PGAdvanceMap(pg=3.13 from=19 to=20)): start 2022-03-29T17:52:32.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=248, detail=PGAdvanceMap(pg=3.14 from=19 to=20)): start 2022-03-29T17:52:32.344 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=249, detail=PGAdvanceMap(pg=3.15 from=19 to=20)): start 2022-03-29T17:52:32.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=250, detail=PGAdvanceMap(pg=3.16 from=19 to=20)): start 2022-03-29T17:52:32.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=251, detail=PGAdvanceMap(pg=3.17 from=19 to=20)): start 2022-03-29T17:52:32.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=252, detail=PGAdvanceMap(pg=3.19 from=19 to=20)): start 2022-03-29T17:52:32.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=253, detail=PGAdvanceMap(pg=3.1a from=19 to=20)): start 2022-03-29T17:52:32.345 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=254, detail=PGAdvanceMap(pg=3.1b from=19 to=20)): start 2022-03-29T17:52:32.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=255, detail=PGAdvanceMap(pg=3.1c from=19 to=20)): start 2022-03-29T17:52:32.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=256, detail=PGAdvanceMap(pg=3.1e from=19 to=20)): start 2022-03-29T17:52:32.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_advance_map(id=257, detail=PGAdvanceMap(pg=3.1f from=19 to=20)): start 2022-03-29T17:52:32.346 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,216 [shard 0] osd - pg_epoch 19 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.347 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 19 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.348 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 19 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.349 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.350 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.351 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,217 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:13 2022-03-29T17:52:32.352 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.955781 3 2022-03-29T17:52:31.290547+0000 2022-03-29T17:52:32.352 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,218 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2622924, 1648576351.2905474 spent on 3 events 2022-03-29T17:52:32.352 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.353 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.956589 0 0.000000 2022-03-29T17:52:32.353 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,218 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2616003, 0.0 spent on 0 events 2022-03-29T17:52:32.353 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.353 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,218 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.914321899s (sent 174.914321899s) 2022-03-29T17:52:32.354 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.355 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.356 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.390 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,218 [shard 0] osd - osd.2: committed_osd_maps(20, 20) 2022-03-29T17:52:32.390 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,218 [shard 0] osd - osd.2: map e 20 marked me up: up_from 7, bind_epoch 0, state active 2022-03-29T17:52:32.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=105, detail=PGAdvanceMap(pg=2.0 from=19 to=20)): start 2022-03-29T17:52:32.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=106, detail=PGAdvanceMap(pg=2.1 from=19 to=20)): start 2022-03-29T17:52:32.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=107, detail=PGAdvanceMap(pg=2.3 from=19 to=20)): start 2022-03-29T17:52:32.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=108, detail=PGAdvanceMap(pg=3.0 from=19 to=20)): start 2022-03-29T17:52:32.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=109, detail=PGAdvanceMap(pg=3.1 from=19 to=20)): start 2022-03-29T17:52:32.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=110, detail=PGAdvanceMap(pg=3.2 from=19 to=20)): start 2022-03-29T17:52:32.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=111, detail=PGAdvanceMap(pg=3.3 from=19 to=20)): start 2022-03-29T17:52:32.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=112, detail=PGAdvanceMap(pg=3.4 from=19 to=20)): start 2022-03-29T17:52:32.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=113, detail=PGAdvanceMap(pg=3.5 from=19 to=20)): start 2022-03-29T17:52:32.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=114, detail=PGAdvanceMap(pg=3.7 from=19 to=20)): start 2022-03-29T17:52:32.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=115, detail=PGAdvanceMap(pg=3.8 from=19 to=20)): start 2022-03-29T17:52:32.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=116, detail=PGAdvanceMap(pg=3.9 from=19 to=20)): start 2022-03-29T17:52:32.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=117, detail=PGAdvanceMap(pg=3.e from=19 to=20)): start 2022-03-29T17:52:32.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=118, detail=PGAdvanceMap(pg=3.11 from=19 to=20)): start 2022-03-29T17:52:32.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=119, detail=PGAdvanceMap(pg=3.14 from=19 to=20)): start 2022-03-29T17:52:32.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=120, detail=PGAdvanceMap(pg=3.16 from=19 to=20)): start 2022-03-29T17:52:32.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=121, detail=PGAdvanceMap(pg=3.18 from=19 to=20)): start 2022-03-29T17:52:32.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=122, detail=PGAdvanceMap(pg=3.19 from=19 to=20)): start 2022-03-29T17:52:32.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=123, detail=PGAdvanceMap(pg=3.1a from=19 to=20)): start 2022-03-29T17:52:32.394 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=124, detail=PGAdvanceMap(pg=3.1d from=19 to=20)): start 2022-03-29T17:52:32.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_advance_map(id=125, detail=PGAdvanceMap(pg=3.1e from=19 to=20)): start 2022-03-29T17:52:32.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,219 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.395 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.396 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 19 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:32.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.397 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.398 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.398 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.398 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.398 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:12 2022-03-29T17:52:32.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.399 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 19 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:32.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.400 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,220 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.401 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.402 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.402 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.402 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.402 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.403 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.404 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.405 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.405 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.405 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.405 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:15 2022-03-29T17:52:32.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 20 2022-03-29T17:52:32.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,221 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 19 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:32.406 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.407 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.408 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:12 2022-03-29T17:52:32.408 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 1.006603 3 2022-03-29T17:52:31.246979+0000 2022-03-29T17:52:32.408 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,222 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2158, 1648576351.2469797 spent on 3 events 2022-03-29T17:52:32.409 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.409 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 1.007467 0 0.000000 2022-03-29T17:52:32.409 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,222 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2150347, 0.0 spent on 0 events 2022-03-29T17:52:32.409 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.409 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,222 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.410 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.855697632s (sent 174.855697632s) 2022-03-29T17:52:32.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 2 2022-03-29T17:52:32.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.411 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.412 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.412 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.412 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.412 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.413 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.413 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,222 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:13 2022-03-29T17:52:32.413 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.413 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.413 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,223 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.414 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 19 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:32.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.415 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.416 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:12 2022-03-29T17:52:32.416 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.992109 3 2022-03-29T17:52:31.248917+0000 2022-03-29T17:52:32.417 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,223 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2315207, 1648576351.248917 spent on 3 events 2022-03-29T17:52:32.417 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.417 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.993969 0 0.000000 2022-03-29T17:52:32.417 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,223 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2297328, 0.0 spent on 0 events 2022-03-29T17:52:32.417 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,223 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.418 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.418 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.418 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.418 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.856918335s (sent 174.856918335s) 2022-03-29T17:52:32.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 2 2022-03-29T17:52:32.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,223 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.419 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.420 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.421 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.421 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:13 2022-03-29T17:52:32.421 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.421 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.422 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,224 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.422 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.423 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.424 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.424 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.984796 3 2022-03-29T17:52:31.249750+0000 2022-03-29T17:52:32.424 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,224 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2398574, 1648576351.2497504 spent on 3 events 2022-03-29T17:52:32.425 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.425 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.986138 0 0.000000 2022-03-29T17:52:32.425 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,224 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2386098, 0.0 spent on 0 events 2022-03-29T17:52:32.425 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.425 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,224 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.426 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.857940674s (sent 174.857940674s) 2022-03-29T17:52:32.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 2 2022-03-29T17:52:32.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,224 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.427 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.428 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.429 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.429 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.429 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,225 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.430 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:12 2022-03-29T17:52:32.431 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.432 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.433 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.433 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.433 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,225 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:12 2022-03-29T17:52:32.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.434 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 19 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.435 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active state: Started advmap 2022-03-29T17:52:32.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=2 pgid=3.1d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.436 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.437 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 1.016398 3 2022-03-29T17:52:31.246839+0000 2022-03-29T17:52:32.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,226 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2102053, 1648576351.2468393 spent on 3 events 2022-03-29T17:52:32.438 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 1.017379 0 0.000000 2022-03-29T17:52:32.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,226 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2093039, 0.0 spent on 0 events 2022-03-29T17:52:32.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.439 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,226 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.439 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.859909058s (sent 174.859909058s) 2022-03-29T17:52:32.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 2 2022-03-29T17:52:32.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,226 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.440 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.441 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.442 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.442 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.443 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,227 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.443 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 19 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.444 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.444 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.444 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.444 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.444 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.445 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.445 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.445 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.445 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:12 2022-03-29T17:52:32.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.446 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,227 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active activate_map 2022-03-29T17:52:32.447 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 luod=0'0 lua=0'0 crt=17'2 mlcod 17'2 active activate_map: Not dirtying info: last_persisted is 17 while current is 20 2022-03-29T17:52:32.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 19 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:32.448 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.449 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 19 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering handle_advance_map {2}/{2} -- 2/2 2022-03-29T17:52:32.450 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,228 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.451 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.452 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.992563 3 2022-03-29T17:52:31.251601+0000 2022-03-29T17:52:32.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,229 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.236518, 1648576351.251602 spent on 3 events 2022-03-29T17:52:32.453 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.453 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering 0.994430 0 0.000000 2022-03-29T17:52:32.453 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,229 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2347276, 0.0 spent on 0 events 2022-03-29T17:52:32.453 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.453 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,229 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown activate - no missing, moving last_complete 17'1 -> 17'1 2022-03-29T17:52:32.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.862380981s (sent 174.862380981s) 2022-03-29T17:52:32.454 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown activate setting up missing_loc from shard 2 2022-03-29T17:52:32.455 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.455 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.455 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.455 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.456 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.456 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.456 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.456 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.456 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.457 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.457 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.457 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.457 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,229 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.458 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,229 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:15 2022-03-29T17:52:32.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 20 2022-03-29T17:52:32.459 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=2 pgid=3.1d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.460 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.461 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.461 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.461 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.461 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.461 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.462 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering publish_stats_to_osd 19:12 2022-03-29T17:52:32.462 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 1.006951 3 2022-03-29T17:52:31.251491+0000 2022-03-29T17:52:32.462 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,230 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.223566, 1648576351.2514913 spent on 3 events 2022-03-29T17:52:32.462 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.463 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 peering exit Started/Primary/Peering 1.008939 0 0.000000 2022-03-29T17:52:32.463 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,230 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2216692, 0.0 spent on 0 events 2022-03-29T17:52:32.463 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.463 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,230 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.463 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown activate - no missing, moving last_complete 18'1 -> 18'1 2022-03-29T17:52:32.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.863830566s (sent 174.863830566s) 2022-03-29T17:52:32.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown activate setting up missing_loc from shard 2 2022-03-29T17:52:32.464 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 1 missing 0 2022-03-29T17:52:32.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {2=1} 2022-03-29T17:52:32.465 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,230 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:32.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized update_calc_stats degraded 1 2022-03-29T17:52:32.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.466 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded publish_stats_to_osd 20:13 2022-03-29T17:52:32.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded state: Activate Finished 2022-03-29T17:52:32.467 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded enter Started/Primary/Active/Activating 2022-03-29T17:52:32.467 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,231 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.467 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 20 2022-03-29T17:52:32.468 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.469 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,218 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.470 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:14 2022-03-29T17:52:32.471 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.471 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.472 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,231 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.472 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,231 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.472 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.472 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.472 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.473 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.474 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.474 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.474 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.474 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: Active advmap 2022-03-29T17:52:32.475 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: Started advmap 2022-03-29T17:52:32.475 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.477 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.477 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.964308 3 2022-03-29T17:52:31.290717+0000 2022-03-29T17:52:32.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,232 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2685418, 1648576351.2907178 spent on 3 events 2022-03-29T17:52:32.478 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,232 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering 0.965131 0 0.000000 2022-03-29T17:52:32.478 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,232 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2677972, 0.0 spent on 0 events 2022-03-29T17:52:32.479 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.479 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,233 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown activate - no missing, moving last_complete 18'3 -> 18'3 2022-03-29T17:52:32.479 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.929031372s (sent 174.929031372s) 2022-03-29T17:52:32.480 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.481 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.482 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.483 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.485 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.486 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,233 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.487 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.487 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.487 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,234 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.488 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean activate_map 2022-03-29T17:52:32.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.490 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean state: handle ActMap primary 2022-03-29T17:52:32.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean publish_stats_to_osd 19:12 2022-03-29T17:52:32.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean activate_map: Not dirtying info: last_persisted is 18 while current is 20 2022-03-29T17:52:32.491 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:12 2022-03-29T17:52:32.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 19 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.495 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,234 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 19 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.496 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.497 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_advance_map(id=124, detail=PGAdvanceMap(pg=3.1d from=19 to=20)): complete 2022-03-29T17:52:32.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=2 pgid=3.18 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=2 pgid=3.18 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_advance_map(id=115, detail=PGAdvanceMap(pg=3.8 from=19 to=20)): complete 2022-03-29T17:52:32.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.500 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:13 2022-03-29T17:52:32.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - pg_epoch 20 pg[3.0( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.501 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.502 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,235 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.502 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 19 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.502 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.502 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.502 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.503 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_advance_map(id=125, detail=PGAdvanceMap(pg=3.1e from=19 to=20)): complete 2022-03-29T17:52:32.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_advance_map(id=118, detail=PGAdvanceMap(pg=3.11 from=19 to=20)): complete 2022-03-29T17:52:32.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_advance_map(id=121, detail=PGAdvanceMap(pg=3.18 from=19 to=20)): complete 2022-03-29T17:52:32.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_advance_map(id=122, detail=PGAdvanceMap(pg=3.19 from=19 to=20)): complete 2022-03-29T17:52:32.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=2 pgid=3.1 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=2 pgid=3.1 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=2 pgid=3.9 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.507 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 19 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.507 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.507 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.507 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.508 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - peering_event(id=627, detail=PeeringEvent(from=1 pgid=3.a sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.508 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 19 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.508 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,236 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.508 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 19 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active state: Started advmap 2022-03-29T17:52:32.509 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=2 pgid=3.9 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.1d 2022-03-29T17:52:32.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 2 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:32.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:32.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.870452881s (sent 174.870452881s) 2022-03-29T17:52:32.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.870666504s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:32.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 19 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.516 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.517 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.518 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 1.010856 3 2022-03-29T17:52:31.291348+0000 2022-03-29T17:52:32.518 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,238 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2272365, 1648576351.2913485 spent on 3 events 2022-03-29T17:52:32.518 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.518 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 1.012828 0 0.000000 2022-03-29T17:52:32.519 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,238 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2253444, 0.0 spent on 0 events 2022-03-29T17:52:32.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,237 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:14 2022-03-29T17:52:32.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:32.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:32.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=2 pgid=3.1d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:32.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.527 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.527 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,238 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.527 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.528 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.528 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.528 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.934783936s (sent 174.934783936s) 2022-03-29T17:52:32.528 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,238 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.529 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.530 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.531 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:15 2022-03-29T17:52:32.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=2 pgid=3.1d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:32.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.534 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.534 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,239 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.534 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=2 pgid=3.e sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_advance_map(id=109, detail=PGAdvanceMap(pg=3.1 from=19 to=20)): complete 2022-03-29T17:52:32.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=2 pgid=3.e sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - pg_advance_map(id=120, detail=PGAdvanceMap(pg=3.16 from=19 to=20)): complete 2022-03-29T17:52:32.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,239 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=2 pgid=3.1d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:32.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.18 2022-03-29T17:52:32.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 2 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:32.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:32.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.873184204s (sent 174.873184204s) 2022-03-29T17:52:32.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 19 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.539 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.540 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:16 2022-03-29T17:52:32.541 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.967721 3 2022-03-29T17:52:31.289181+0000 2022-03-29T17:52:32.541 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,240 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2728937, 1648576351.2891817 spent on 3 events 2022-03-29T17:52:32.541 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.541 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.968447 0 0.000000 2022-03-29T17:52:32.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.873931885s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:32.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,240 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:14 2022-03-29T17:52:32.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:32.550 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,241 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2722387, 0.0 spent on 0 events 2022-03-29T17:52:32.550 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.551 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,241 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.551 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.937301636s (sent 174.937301636s) 2022-03-29T17:52:32.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.552 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.553 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.554 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:32.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=2 pgid=3.18 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:32.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:15 2022-03-29T17:52:32.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=2 pgid=3.18 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:32.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:17 2022-03-29T17:52:32.560 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.561 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.561 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,242 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.561 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.562 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering publish_stats_to_osd 19:13 2022-03-29T17:52:32.563 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.985622 3 2022-03-29T17:52:31.289552+0000 2022-03-29T17:52:32.563 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,242 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2569678, 1648576351.2895525 spent on 3 events 2022-03-29T17:52:32.563 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.563 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,242 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 peering exit Started/Primary/Peering 0.986388 0 0.000000 2022-03-29T17:52:32.564 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,242 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2562737, 0.0 spent on 0 events 2022-03-29T17:52:32.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=2 pgid=3.3 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_advance_map(id=110, detail=PGAdvanceMap(pg=3.2 from=19 to=20)): complete 2022-03-29T17:52:32.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,242 [shard 0] osd - pg_advance_map(id=116, detail=PGAdvanceMap(pg=3.9 from=19 to=20)): complete 2022-03-29T17:52:32.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=2 pgid=3.18 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:32.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_advance_map(id=117, detail=PGAdvanceMap(pg=3.e from=19 to=20)): complete 2022-03-29T17:52:32.567 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.568 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,243 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown activate - no missing, moving last_complete 18'3 -> 18'3 2022-03-29T17:52:32.568 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.939285278s (sent 174.939285278s) 2022-03-29T17:52:32.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.569 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:32.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:32.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:32.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats degraded 2 2022-03-29T17:52:32.570 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.571 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded publish_stats_to_osd 20:14 2022-03-29T17:52:32.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_advance_map(id=105, detail=PGAdvanceMap(pg=2.0 from=19 to=20)): complete 2022-03-29T17:52:32.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=2 pgid=3.3 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_advance_map(id=106, detail=PGAdvanceMap(pg=2.1 from=19 to=20)): complete 2022-03-29T17:52:32.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,243 [shard 0] osd - pg_advance_map(id=123, detail=PGAdvanceMap(pg=3.1a from=19 to=20)): complete 2022-03-29T17:52:32.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_advance_map(id=111, detail=PGAdvanceMap(pg=3.3 from=19 to=20)): complete 2022-03-29T17:52:32.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_advance_map(id=107, detail=PGAdvanceMap(pg=2.3 from=19 to=20)): complete 2022-03-29T17:52:32.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.e 2022-03-29T17:52:32.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded state: _activate_committed 20 peer_activated now 2 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:32.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded state: all_activated_and_committed 2022-03-29T17:52:32.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded recalc_readable_until readable_until[_ub] 174.877197266s (sent 174.877197266s) 2022-03-29T17:52:32.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded state: Activate Finished 2022-03-29T17:52:32.576 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded enter Started/Primary/Active/Activating 2022-03-29T17:52:32.576 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,244 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.576 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 19 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.577 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.578 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded update_calc_stats shard 2 primary objects 1 missing 0 2022-03-29T17:52:32.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded update_calc_stats object_location_counts {2=1} 2022-03-29T17:52:32.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:32.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded update_calc_stats degraded 1 2022-03-29T17:52:32.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 activating+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:32.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded state: mnow 158.877990723s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:32.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded share_pg_info 2022-03-29T17:52:32.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats shard 2 primary objects 1 missing 0 2022-03-29T17:52:32.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {2=1} 2022-03-29T17:52:32.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:32.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 1 2022-03-29T17:52:32.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:32.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:14 2022-03-29T17:52:32.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 19 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.585 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.586 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded needs_recovery is recovered 2022-03-29T17:52:32.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded needs_backfill does not need backfill 2022-03-29T17:52:32.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:32.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=2 pgid=3.e sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:32.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats shard 2 primary objects 1 missing 0 2022-03-29T17:52:32.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 19 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.589 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,245 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.590 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.591 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.591 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.971977 3 2022-03-29T17:52:31.293813+0000 2022-03-29T17:52:32.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {2=1} 2022-03-29T17:52:32.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:32.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 1 2022-03-29T17:52:32.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:32.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:15 2022-03-29T17:52:32.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=2 pgid=3.e sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:32.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_advance_map(id=113, detail=PGAdvanceMap(pg=3.5 from=19 to=20)): complete 2022-03-29T17:52:32.597 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,246 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.274186, 1648576351.2938135 spent on 3 events 2022-03-29T17:52:32.598 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.598 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.973181 0 0.000000 2022-03-29T17:52:32.598 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,246 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2735405, 0.0 spent on 0 events 2022-03-29T17:52:32.598 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.599 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,246 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.599 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.942794800s (sent 174.942794800s) 2022-03-29T17:52:32.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,246 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.600 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.601 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.602 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_advance_map(id=108, detail=PGAdvanceMap(pg=3.0 from=19 to=20)): complete 2022-03-29T17:52:32.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_advance_map(id=114, detail=PGAdvanceMap(pg=3.7 from=19 to=20)): complete 2022-03-29T17:52:32.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_advance_map(id=119, detail=PGAdvanceMap(pg=3.14 from=19 to=20)): complete 2022-03-29T17:52:32.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - peering_event(id=328, detail=PeeringEvent(from=2 pgid=3.1d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:32.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_advance_map(id=112, detail=PGAdvanceMap(pg=3.4 from=19 to=20)): complete 2022-03-29T17:52:32.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.1d 2022-03-29T17:52:32.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.024585 2 0.002771 2022-03-29T17:52:32.605 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,247 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.223038, 0.002771924 spent on 2 events 2022-03-29T17:52:32.605 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.607 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:32.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,247 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:32.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000158 0 0.000000 2022-03-29T17:52:32.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,247 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.247689, 0.0 spent on 0 events 2022-03-29T17:52:32.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:32.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,247 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:32.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,247 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:16 2022-03-29T17:52:32.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=2 pgid=3.1d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:32.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.613 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.614 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.614 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,248 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.614 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 20 2022-03-29T17:52:32.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.615 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.616 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering publish_stats_to_osd 19:13 2022-03-29T17:52:32.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.9 2022-03-29T17:52:32.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 2 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:32.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:32.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.881896973s (sent 174.881896973s) 2022-03-29T17:52:32.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,248 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized state: mnow 158.882110596s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:32.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.624 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.973908 3 2022-03-29T17:52:31.293326+0000 2022-03-29T17:52:32.624 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,249 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2753386, 1648576351.2933261 spent on 3 events 2022-03-29T17:52:32.624 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.624 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 peering exit Started/Primary/Peering 0.974643 0 0.000000 2022-03-29T17:52:32.625 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,249 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.274695, 0.0 spent on 0 events 2022-03-29T17:52:32.625 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.625 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,249 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.625 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown activate - no missing, moving last_complete 18'4 -> 18'4 2022-03-29T17:52:32.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.945434570s (sent 174.945434570s) 2022-03-29T17:52:32.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.626 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 1 missing 0 2022-03-29T17:52:32.627 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {1=1} 2022-03-29T17:52:32.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:32.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:32.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:32.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=2 pgid=3.9 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:32.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:32.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,249 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:32.635 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized update_calc_stats degraded 1 2022-03-29T17:52:32.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded publish_stats_to_osd 20:14 2022-03-29T17:52:32.636 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded state: Activate Finished 2022-03-29T17:52:32.637 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded enter Started/Primary/Active/Activating 2022-03-29T17:52:32.637 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,250 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.637 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=2 pgid=3.9 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:32.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.1 2022-03-29T17:52:32.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 2 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:32.640 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 19 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.641 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.642 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,250 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:32.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.884307861s (sent 174.884307861s) 2022-03-29T17:52:32.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.884521484s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:32.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.650 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.651 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.991099 3 2022-03-29T17:52:31.291027+0000 2022-03-29T17:52:32.651 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,251 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2606916, 1648576351.2910278 spent on 3 events 2022-03-29T17:52:32.651 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.651 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.991897 0 0.000000 2022-03-29T17:52:32.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,251 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2599728, 0.0 spent on 0 events 2022-03-29T17:52:32.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.652 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,251 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.652 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,251 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.947921753s (sent 174.947921753s) 2022-03-29T17:52:32.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.653 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.654 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.655 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:32.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:32.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:32.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=2 pgid=3.1 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:32.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.658 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.659 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.659 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.659 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,252 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.660 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:32.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,252 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=2 pgid=3.1 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:32.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=2 pgid=3.e sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:32.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.666 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:13 2022-03-29T17:52:32.667 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - pg_epoch 20 pg[3.19( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - peering_event(id=627, detail=PeeringEvent(from=1 pgid=3.a sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.668 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.669 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.670 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,253 [shard 0] osd - osd.2: now active 2022-03-29T17:52:32.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=2 pgid=3.9 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:32.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,253 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=2 pgid=3.1 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:32.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - peering_event(id=329, detail=PeeringEvent(from=2 pgid=3.18 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:32.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.3 2022-03-29T17:52:32.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 2 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:32.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:32.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 19 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.673 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 19 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.674 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 19 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Active advmap 2022-03-29T17:52:32.675 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Started advmap 2022-03-29T17:52:32.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.887207031s (sent 174.887207031s) 2022-03-29T17:52:32.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.887969971s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:32.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,254 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:32.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active activate_map 2022-03-29T17:52:32.683 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 luod=0'0 lua=0'0 crt=18'1 mlcod 18'1 active activate_map: Not dirtying info: last_persisted is 18 while current is 20 2022-03-29T17:52:32.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.684 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:13 2022-03-29T17:52:32.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.4( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.685 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 19 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.686 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:32.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:32.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=2 pgid=3.3 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:32.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,255 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:32.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=2 pgid=3.3 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:32.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.693 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 19 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.694 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.695 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.18 2022-03-29T17:52:32.696 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.032425 2 0.002652 2022-03-29T17:52:32.696 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,256 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2241964, 0.002652578 spent on 2 events 2022-03-29T17:52:32.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.698 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:13 2022-03-29T17:52:32.699 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 1.002972 3 2022-03-29T17:52:31.290156+0000 2022-03-29T17:52:32.699 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,256 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2538452, 1648576351.2901568 spent on 3 events 2022-03-29T17:52:32.699 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.699 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 1.003716 0 0.000000 2022-03-29T17:52:32.700 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,256 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2531781, 0.0 spent on 0 events 2022-03-29T17:52:32.700 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.700 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,256 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,256 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.700 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.952972412s (sent 174.952972412s) 2022-03-29T17:52:32.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.701 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.702 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.704 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:32.704 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,257 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:32.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.705 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000605 0 0.000000 2022-03-29T17:52:32.705 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,257 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.2567065, 0.0 spent on 0 events 2022-03-29T17:52:32.705 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:32.706 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,257 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:32.706 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.706 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.706 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.706 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.708 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.708 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:16 2022-03-29T17:52:32.708 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.708 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=2 pgid=3.18 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:32.708 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.710 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.711 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:14 2022-03-29T17:52:32.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.712 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,257 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.712 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,257 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.713 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.715 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=2 pgid=3.3 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:32.715 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.e 2022-03-29T17:52:32.715 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded exit Started/Primary/Active/Activating 0.027323 2 0.003146 2022-03-29T17:52:32.715 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.231173, 0.003146905 spent on 2 events 2022-03-29T17:52:32.716 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded enter Started/Primary/Active/Recovered 2022-03-29T17:52:32.716 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:32.716 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded needs_recovery is recovered 2022-03-29T17:52:32.716 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded exit Started/Primary/Active/Recovered 0.000063 0 0.000000 2022-03-29T17:52:32.716 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.2585592, 0.0 spent on 0 events 2022-03-29T17:52:32.717 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded enter Started/Primary/Active/Clean 2022-03-29T17:52:32.717 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:32.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.719 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:16 2022-03-29T17:52:32.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.999517 3 2022-03-29T17:52:31.293228+0000 2022-03-29T17:52:32.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2593048, 1648576351.2932286 spent on 3 events 2022-03-29T17:52:32.720 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.720 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 1.000259 0 0.000000 2022-03-29T17:52:32.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2586398, 0.0 spent on 0 events 2022-03-29T17:52:32.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.721 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,258 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.721 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,258 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.954925537s (sent 174.954925537s) 2022-03-29T17:52:32.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.722 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.723 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.724 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded share_pg_info 2022-03-29T17:52:32.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats shard 2 primary objects 1 missing 0 2022-03-29T17:52:32.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {2=1} 2022-03-29T17:52:32.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:32.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 1 2022-03-29T17:52:32.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:32.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:16 2022-03-29T17:52:32.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=2 pgid=3.e sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:32.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.728 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.729 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - peering_event(id=330, detail=PeeringEvent(from=2 pgid=3.1 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:32.729 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - peering_event(id=333, detail=PeeringEvent(from=2 pgid=3.e sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:32.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - peering_event(id=331, detail=PeeringEvent(from=2 pgid=3.9 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:32.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,259 [shard 0] osd - peering_event(id=335, detail=PeeringEvent(from=2 pgid=3.3 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:32.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - peering_event(id=332, detail=PeeringEvent(from=2 pgid=3.1d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): complete 2022-03-29T17:52:32.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.1 2022-03-29T17:52:32.731 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.034761 2 0.002780 2022-03-29T17:52:32.731 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2252958, 0.00278028 spent on 2 events 2022-03-29T17:52:32.731 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:32.731 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:32.731 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.733 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:17 2022-03-29T17:52:32.734 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.735 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.735 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.735 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.738 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000075 0 0.000000 2022-03-29T17:52:32.738 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.2601166, 0.0 spent on 0 events 2022-03-29T17:52:32.738 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:32.738 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,260 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.740 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.740 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.740 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.740 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.741 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:8 2022-03-29T17:52:32.741 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.741 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=2 pgid=3.1 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:32.741 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.741 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.9 2022-03-29T17:52:32.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.745 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:16 2022-03-29T17:52:32.746 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.976107 3 2022-03-29T17:52:31.293715+0000 2022-03-29T17:52:32.746 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,261 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2852588, 1648576351.2937152 spent on 3 events 2022-03-29T17:52:32.746 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.747 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.976799 0 0.000000 2022-03-29T17:52:32.747 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,261 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2846396, 0.0 spent on 0 events 2022-03-29T17:52:32.747 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.747 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,261 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.957473755s (sent 174.957473755s) 2022-03-29T17:52:32.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.748 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.749 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.750 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:17 2022-03-29T17:52:32.751 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.751 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.751 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,261 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.751 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,261 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean activate_map 2022-03-29T17:52:32.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.752 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean state: handle ActMap primary 2022-03-29T17:52:32.753 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.757 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.031490 2 0.002178 2022-03-29T17:52:32.757 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2297008, 0.002178675 spent on 2 events 2022-03-29T17:52:32.757 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:32.757 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:32.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.758 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000067 0 0.000000 2022-03-29T17:52:32.758 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.26228, 0.0 spent on 0 events 2022-03-29T17:52:32.758 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:32.759 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:32.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized publish_stats_to_osd 20:8 2022-03-29T17:52:32.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=2 pgid=3.9 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:32.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - peering_event(id=334, detail=PeeringEvent(from=2 pgid=3.18 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): complete 2022-03-29T17:52:32.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - peering_event(id=336, detail=PeeringEvent(from=2 pgid=3.e sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): complete 2022-03-29T17:52:32.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,262 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.3 2022-03-29T17:52:32.762 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.035762 2 0.002662 2022-03-29T17:52:32.763 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,262 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.227205, 0.002662607 spent on 2 events 2022-03-29T17:52:32.763 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:32.763 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:32.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.763 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000052 0 0.000000 2022-03-29T17:52:32.764 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.263014, 0.0 spent on 0 events 2022-03-29T17:52:32.764 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:32.764 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:32.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 2 upset 2 acting_recovery_backfill 2 2022-03-29T17:52:32.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 2 primary objects 0 missing 0 2022-03-29T17:52:32.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean publish_stats_to_osd 19:13 2022-03-29T17:52:32.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.1a( v 17'2 (0'0,17'2] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 lua=0'0 crt=17'2 mlcod 17'2 active+clean activate_map: Not dirtying info: last_persisted is 17 while current is 20 2022-03-29T17:52:32.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.767 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.768 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:32.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.768 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering publish_stats_to_osd 19:19 2022-03-29T17:52:32.769 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 1.012342 3 2022-03-29T17:52:31.290774+0000 2022-03-29T17:52:32.769 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.251213, 1648576351.2907743 spent on 3 events 2022-03-29T17:52:32.769 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.769 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 peering exit Started/Primary/Peering 1.013081 0 0.000000 2022-03-29T17:52:32.770 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2505503, 0.0 spent on 0 events 2022-03-29T17:52:32.770 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.770 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,263 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.771 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 2 missing= 0 2022-03-29T17:52:32.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:8 2022-03-29T17:52:32.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=2 pgid=3.3 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:32.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - peering_event(id=337, detail=PeeringEvent(from=2 pgid=3.9 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): complete 2022-03-29T17:52:32.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - peering_event(id=338, detail=PeeringEvent(from=2 pgid=3.1 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): complete 2022-03-29T17:52:32.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,263 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=8/9 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.776 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown activate - no missing, moving last_complete 12'97 -> 12'97 2022-03-29T17:52:32.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.960205078s (sent 174.960205078s) 2022-03-29T17:52:32.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.777 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:32.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:32.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:32.778 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized update_calc_stats degraded 2 2022-03-29T17:52:32.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded publish_stats_to_osd 20:20 2022-03-29T17:52:32.779 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded state: Activate Finished 2022-03-29T17:52:32.780 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded enter Started/Primary/Active/Activating 2022-03-29T17:52:32.780 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,264 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.780 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.781 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:32,264 [shard 0] osd - peering_event(id=339, detail=PeeringEvent(from=2 pgid=3.3 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): complete 2022-03-29T17:52:32.908 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 19 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.909 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 19 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.910 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 19 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.911 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 19 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.912 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,265 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.913 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.914 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.986817 3 2022-03-29T17:52:31.294861+0000 2022-03-29T17:52:32.914 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,266 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.279244, 1648576351.2948613 spent on 3 events 2022-03-29T17:52:32.914 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.914 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.987536 0 0.000000 2022-03-29T17:52:32.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,266 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.278597, 0.0 spent on 0 events 2022-03-29T17:52:32.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.915 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,266 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.915 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.962203979s (sent 174.962203979s) 2022-03-29T17:52:32.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.916 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.917 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.918 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.919 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.919 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,266 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.919 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 19 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.920 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,266 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.921 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.922 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 1.002010 3 2022-03-29T17:52:31.290831+0000 2022-03-29T17:52:32.922 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2651584, 1648576351.2908316 spent on 3 events 2022-03-29T17:52:32.922 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 peering exit Started/Primary/Peering 1.002762 0 0.000000 2022-03-29T17:52:32.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2645001, 0.0 spent on 0 events 2022-03-29T17:52:32.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.923 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown activate - no missing, moving last_complete 17'1 -> 17'1 2022-03-29T17:52:32.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.963333130s (sent 174.963333130s) 2022-03-29T17:52:32.924 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.925 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.926 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.927 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.927 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.927 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,267 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.928 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,267 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:32.928 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.929 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 19 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 19 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.930 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - peering_event(id=628, detail=PeeringEvent(from=1 pgid=3.b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.931 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.932 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.932 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,268 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.998170 3 2022-03-29T17:52:31.289775+0000 2022-03-29T17:52:32.932 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,268 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2707744, 1648576351.2897751 spent on 3 events 2022-03-29T17:52:32.933 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.933 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 peering exit Started/Primary/Peering 0.998958 0 0.000000 2022-03-29T17:52:32.933 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,269 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2700555, 0.0 spent on 0 events 2022-03-29T17:52:32.933 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.934 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,269 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown activate - no missing, moving last_complete 18'5 -> 18'5 2022-03-29T17:52:32.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.934 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.965072632s (sent 174.965072632s) 2022-03-29T17:52:32.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.935 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.937 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.937 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.938 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,269 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.938 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 19 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 19 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.939 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 19 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering handle_advance_map {1}/{1} -- 1/1 2022-03-29T17:52:32.940 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,269 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Peering advmap 2022-03-29T17:52:32.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering adjust_need_up_thru now 19, need_up_thru now false 2022-03-29T17:52:32.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Started advmap 2022-03-29T17:52:32.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering check_recovery_sources no source osds () went down 2022-03-29T17:52:32.941 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.942 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.942 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.992919 3 2022-03-29T17:52:31.291514+0000 2022-03-29T17:52:32.942 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,270 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2772658, 1648576351.291514 spent on 3 events 2022-03-29T17:52:32.943 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.943 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.993676 0 0.000000 2022-03-29T17:52:32.943 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,270 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2766058, 0.0 spent on 0 events 2022-03-29T17:52:32.943 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.944 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,270 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.944 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.966323853s (sent 174.966323853s) 2022-03-29T17:52:32.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.945 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.946 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.947 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.947 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,270 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 20 pg[3.6( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.948 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,270 [shard 0] osd - pg_epoch 19 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - peering_event(id=628, detail=PeeringEvent(from=1 pgid=3.b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.949 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.950 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:13 2022-03-29T17:52:32.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[3.2( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.951 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.952 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.990323 3 2022-03-29T17:52:31.295040+0000 2022-03-29T17:52:32.953 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,271 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2813697, 1648576351.2950404 spent on 3 events 2022-03-29T17:52:32.953 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.991053 0 0.000000 2022-03-29T17:52:32.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,271 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2807236, 0.0 spent on 0 events 2022-03-29T17:52:32.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.954 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,271 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.967819214s (sent 174.967819214s) 2022-03-29T17:52:32.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.955 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,271 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.956 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.957 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.958 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.958 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,272 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.958 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[2.2( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 19 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean handle_advance_map {1, 2}/{1, 2} -- 1/1 2022-03-29T17:52:32.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active advmap 2022-03-29T17:52:32.959 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Started advmap 2022-03-29T17:52:32.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean check_recovery_sources no source osds () went down 2022-03-29T17:52:32.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.960 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:13 2022-03-29T17:52:32.961 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.992492 3 2022-03-29T17:52:31.291880+0000 2022-03-29T17:52:32.961 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,272 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.280286, 1648576351.2918808 spent on 3 events 2022-03-29T17:52:32.961 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.961 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.993204 0 0.000000 2022-03-29T17:52:32.962 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,272 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2796614, 0.0 spent on 0 events 2022-03-29T17:52:32.962 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.962 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,272 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.962 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,272 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.968917847s (sent 174.968917847s) 2022-03-29T17:52:32.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.963 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.964 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.965 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:14 2022-03-29T17:52:32.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.966 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.966 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,273 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.966 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=1 lpr=11 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 12 while current is 20 2022-03-29T17:52:32.967 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.968 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 19 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active handle_advance_map {2, 1}/{2, 1} -- 2/2 2022-03-29T17:52:32.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active state: Started advmap 2022-03-29T17:52:32.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active check_recovery_sources no source osds () went down 2022-03-29T17:52:32.969 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,273 [shard 0] osd - peering_event(id=629, detail=PeeringEvent(from=1 pgid=2.6 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.970 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:13 2022-03-29T17:52:32.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[3.14( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,2] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.971 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.972 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:16 2022-03-29T17:52:32.972 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 1.025343 3 2022-03-29T17:52:31.293022+0000 2022-03-29T17:52:32.972 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,274 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.2490363, 1648576351.2930229 spent on 3 events 2022-03-29T17:52:32.973 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.973 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 1.027216 0 0.000000 2022-03-29T17:52:32.973 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,274 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2472308, 0.0 spent on 0 events 2022-03-29T17:52:32.973 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.973 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,274 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.970474243s (sent 174.970474243s) 2022-03-29T17:52:32.974 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.975 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.976 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:17 2022-03-29T17:52:32.977 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.977 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.977 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,274 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - pg_epoch 20 pg[2.7( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,274 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - peering_event(id=629, detail=PeeringEvent(from=1 pgid=2.6 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.978 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering activate_map 2022-03-29T17:52:32.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: handle ActMap primary 2022-03-29T17:52:32.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.979 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering publish_stats_to_osd 19:4 2022-03-29T17:52:32.979 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering/WaitUpThru 0.992613 3 2022-03-29T17:52:31.292834+0000 2022-03-29T17:52:32.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,275 [shard 0] osd - Exiting state: Started/Primary/Peering/WaitUpThru, entered at 1648576351.282657, 1648576351.292834 spent on 3 events 2022-03-29T17:52:32.980 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering state: Leaving Peering 2022-03-29T17:52:32.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 peering exit Started/Primary/Peering 0.993318 0 0.000000 2022-03-29T17:52:32.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,275 [shard 0] osd - Exiting state: Started/Primary/Peering, entered at 1648576351.2820168, 0.0 spent on 0 events 2022-03-29T17:52:32.980 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Active 2022-03-29T17:52:32.981 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,275 [shard 0] osd - Entering state: Started/Primary/Active 2022-03-29T17:52:32.981 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown state: In Active, about to call activate 2022-03-29T17:52:32.981 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - no missing, moving last_complete 0'0 -> 0'0 2022-03-29T17:52:32.981 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate - purged_snaps [] removed_snaps [2~1] 2022-03-29T17:52:32.981 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown recalc_readable_until readable_until[_ub] 174.971389771s (sent 174.971389771s) 2022-03-29T17:52:32.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown activate setting up missing_loc from shard 1 2022-03-29T17:52:32.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 unknown needs_recovery is recovered 2022-03-29T17:52:32.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.982 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.983 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized publish_stats_to_osd 20:5 2022-03-29T17:52:32.984 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: Activate Finished 2022-03-29T17:52:32.984 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized enter Started/Primary/Active/Activating 2022-03-29T17:52:32.985 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,275 [shard 0] osd - Entering state: Started/Primary/Active/Activating 2022-03-29T17:52:32.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized activate_map: Not dirtying info: last_persisted is 19 while current is 20 2022-03-29T17:52:32.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,275 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.985 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - peering_event(id=630, detail=PeeringEvent(from=1 pgid=3.12 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - peering_event(id=630, detail=PeeringEvent(from=1 pgid=3.12 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map 2022-03-29T17:52:32.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=1 lpr=14 crt=0'0 mlcod 0'0 active activate_map: Not dirtying info: last_persisted is 15 while current is 20 2022-03-29T17:52:32.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.986 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map 2022-03-29T17:52:32.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: Active: handling ActMap 2022-03-29T17:52:32.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean state: handle ActMap primary 2022-03-29T17:52:32.987 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.988 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,276 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean publish_stats_to_osd 19:16 2022-03-29T17:52:32.988 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - pg_epoch 20 pg[2.3( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [1,2] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean activate_map: Not dirtying info: last_persisted is 12 while current is 20 2022-03-29T17:52:32.988 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - log is not dirty 2022-03-29T17:52:32.988 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:32.988 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - peering_event(id=631, detail=PeeringEvent(from=1 pgid=3.d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.989 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - pg_advance_map(id=230, detail=PGAdvanceMap(pg=2.6 from=19 to=20)): complete 2022-03-29T17:52:32.989 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - peering_event(id=632, detail=PeeringEvent(from=1 pgid=3.10 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.989 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - peering_event(id=631, detail=PeeringEvent(from=1 pgid=3.d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.989 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - peering_event(id=632, detail=PeeringEvent(from=1 pgid=3.10 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.989 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - pg_advance_map(id=239, detail=PGAdvanceMap(pg=3.a from=19 to=20)): complete 2022-03-29T17:52:32.990 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - peering_event(id=633, detail=PeeringEvent(from=1 pgid=2.4 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.990 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - pg_advance_map(id=232, detail=PGAdvanceMap(pg=3.0 from=19 to=20)): complete 2022-03-29T17:52:32.990 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,277 [shard 0] osd - pg_advance_map(id=225, detail=PGAdvanceMap(pg=2.1 from=19 to=20)): complete 2022-03-29T17:52:32.990 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,277 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:32.991 INFO:tasks.ceph.osd.1.smithi138.stderr:WARN 2022-03-29 17:52:32,278 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:52:32.991 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_advance_map(id=240, detail=PGAdvanceMap(pg=3.b from=19 to=20)): complete 2022-03-29T17:52:32.991 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,278 [shard 0] ms - [osd.1(cluster) v2:172.21.15.138:6801/35578 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:32.991 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_advance_map(id=237, detail=PGAdvanceMap(pg=3.7 from=19 to=20)): complete 2022-03-29T17:52:32.991 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - peering_event(id=634, detail=PeeringEvent(from=1 pgid=3.1f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.992 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_advance_map(id=246, detail=PGAdvanceMap(pg=3.12 from=19 to=20)): complete 2022-03-29T17:52:32.992 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - peering_event(id=635, detail=PeeringEvent(from=1 pgid=2.5 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:32.992 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_advance_map(id=244, detail=PGAdvanceMap(pg=3.10 from=19 to=20)): complete 2022-03-29T17:52:32.992 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - peering_event(id=634, detail=PeeringEvent(from=1 pgid=3.1f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.993 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - peering_event(id=635, detail=PeeringEvent(from=1 pgid=2.5 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:32.993 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_advance_map(id=245, detail=PGAdvanceMap(pg=3.11 from=19 to=20)): complete 2022-03-29T17:52:32.993 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.a 2022-03-29T17:52:32.993 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:32.993 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:32.994 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.974670410s (sent 174.974670410s) 2022-03-29T17:52:32.994 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.994 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.994 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.995 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.995 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.995 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.995 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.995 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized state: mnow 158.974853516s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:32.996 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,278 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:32.996 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.996 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.996 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:32.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:32.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:32.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:32.997 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:32.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:32.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:32.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:32.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:32.998 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - peering_event(id=636, detail=PeeringEvent(from=1 pgid=3.a sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:32.999 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:32.999 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:32.999 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:32.999 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.000 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.000 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.000 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.000 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.000 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:33.001 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.001 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - peering_event(id=627, detail=PeeringEvent(from=1 pgid=3.a sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.001 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.001 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - peering_event(id=633, detail=PeeringEvent(from=1 pgid=2.4 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.d 2022-03-29T17:52:33.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded state: all_activated_and_committed 2022-03-29T17:52:33.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded recalc_readable_until readable_until[_ub] 174.975784302s (sent 174.975784302s) 2022-03-29T17:52:33.002 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded update_calc_stats shard 1 primary objects 1 missing 0 2022-03-29T17:52:33.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded update_calc_stats object_location_counts {1=1} 2022-03-29T17:52:33.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,279 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.003 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:33.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded update_calc_stats degraded 1 2022-03-29T17:52:33.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 activating+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded state: mnow 158.975967407s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded share_pg_info 2022-03-29T17:52:33.004 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats shard 1 primary objects 1 missing 0 2022-03-29T17:52:33.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {1=1} 2022-03-29T17:52:33.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.005 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:33.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 1 2022-03-29T17:52:33.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:15 2022-03-29T17:52:33.006 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded needs_recovery is recovered 2022-03-29T17:52:33.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded needs_backfill does not need backfill 2022-03-29T17:52:33.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - peering_event(id=637, detail=PeeringEvent(from=1 pgid=3.d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.007 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats shard 1 primary objects 1 missing 0 2022-03-29T17:52:33.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {1=1} 2022-03-29T17:52:33.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:33.008 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 1 2022-03-29T17:52:33.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:16 2022-03-29T17:52:33.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.009 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - peering_event(id=631, detail=PeeringEvent(from=1 pgid=3.d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_advance_map(id=228, detail=PGAdvanceMap(pg=2.4 from=19 to=20)): complete 2022-03-29T17:52:33.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - peering_event(id=636, detail=PeeringEvent(from=1 pgid=3.a sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.010 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - pg_advance_map(id=257, detail=PGAdvanceMap(pg=3.1f from=19 to=20)): complete 2022-03-29T17:52:33.011 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - peering_event(id=638, detail=PeeringEvent(from=1 pgid=3.f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.011 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,280 [shard 0] osd - peering_event(id=638, detail=PeeringEvent(from=1 pgid=3.f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.011 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - peering_event(id=639, detail=PeeringEvent(from=1 pgid=3.1c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.011 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_advance_map(id=229, detail=PGAdvanceMap(pg=2.5 from=19 to=20)): complete 2022-03-29T17:52:33.011 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_advance_map(id=252, detail=PGAdvanceMap(pg=3.19 from=19 to=20)): complete 2022-03-29T17:52:33.012 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - peering_event(id=640, detail=PeeringEvent(from=1 pgid=3.17 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.012 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - peering_event(id=639, detail=PeeringEvent(from=1 pgid=3.1c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.012 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.10 2022-03-29T17:52:33.012 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.012 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.013 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.977355957s (sent 174.977355957s) 2022-03-29T17:52:33.013 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.013 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.013 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.014 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.014 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.014 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.014 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.977539062s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.015 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.016 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:15 2022-03-29T17:52:33.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.017 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - peering_event(id=641, detail=PeeringEvent(from=1 pgid=3.10 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,281 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.018 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.019 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:16 2022-03-29T17:52:33.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - peering_event(id=632, detail=PeeringEvent(from=1 pgid=3.10 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.020 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - peering_event(id=642, detail=PeeringEvent(from=1 pgid=1.0 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 2.6 2022-03-29T17:52:33.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 11 last_epoch_started 12 same_interval_since 19 2022-03-29T17:52:33.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.021 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.978485107s (sent 174.978485107s) 2022-03-29T17:52:33.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.022 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.978668213s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.023 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.024 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.025 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,282 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:18 2022-03-29T17:52:33.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - peering_event(id=643, detail=PeeringEvent(from=1 pgid=2.6 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.026 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.027 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:19 2022-03-29T17:52:33.028 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - peering_event(id=629, detail=PeeringEvent(from=1 pgid=2.6 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.b 2022-03-29T17:52:33.029 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.979400635s (sent 174.979400635s) 2022-03-29T17:52:33.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.030 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.031 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.979598999s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.032 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.033 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:15 2022-03-29T17:52:33.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,283 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.034 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - peering_event(id=644, detail=PeeringEvent(from=1 pgid=3.b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.035 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.036 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:16 2022-03-29T17:52:33.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - peering_event(id=628, detail=PeeringEvent(from=1 pgid=3.b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.037 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - peering_event(id=643, detail=PeeringEvent(from=1 pgid=2.6 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.12 2022-03-29T17:52:33.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.038 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.980484009s (sent 174.980484009s) 2022-03-29T17:52:33.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.039 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.980667114s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.040 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.041 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.042 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,284 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:33.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - peering_event(id=645, detail=PeeringEvent(from=1 pgid=3.12 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.043 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.044 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:33.045 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - peering_event(id=630, detail=PeeringEvent(from=1 pgid=3.12 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - peering_event(id=642, detail=PeeringEvent(from=1 pgid=1.0 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.046 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_advance_map(id=242, detail=PGAdvanceMap(pg=3.d from=19 to=20)): complete 2022-03-29T17:52:33.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - peering_event(id=637, detail=PeeringEvent(from=1 pgid=3.d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_advance_map(id=251, detail=PGAdvanceMap(pg=3.17 from=19 to=20)): complete 2022-03-29T17:52:33.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - peering_event(id=641, detail=PeeringEvent(from=1 pgid=3.10 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.1f 2022-03-29T17:52:33.047 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,285 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.981887817s (sent 174.981887817s) 2022-03-29T17:52:33.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.048 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.049 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.982116699s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.050 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.051 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:33.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.052 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - peering_event(id=646, detail=PeeringEvent(from=1 pgid=3.1f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.053 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.054 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:33.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.055 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - peering_event(id=634, detail=PeeringEvent(from=1 pgid=3.1f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_advance_map(id=255, detail=PGAdvanceMap(pg=3.1c from=19 to=20)): complete 2022-03-29T17:52:33.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - pg_advance_map(id=243, detail=PGAdvanceMap(pg=3.f from=19 to=20)): complete 2022-03-29T17:52:33.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,286 [shard 0] osd - peering_event(id=647, detail=PeeringEvent(from=1 pgid=3.1b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.056 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - peering_event(id=640, detail=PeeringEvent(from=1 pgid=3.17 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - peering_event(id=647, detail=PeeringEvent(from=1 pgid=3.1b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 2.5 2022-03-29T17:52:33.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 11 last_epoch_started 12 same_interval_since 19 2022-03-29T17:52:33.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.057 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.983230591s (sent 174.983230591s) 2022-03-29T17:52:33.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.058 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.059 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.983459473s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.060 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.061 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:18 2022-03-29T17:52:33.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.062 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.063 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - peering_event(id=648, detail=PeeringEvent(from=1 pgid=2.5 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.063 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.063 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.063 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.063 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,287 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.064 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:19 2022-03-29T17:52:33.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - peering_event(id=635, detail=PeeringEvent(from=1 pgid=2.5 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.065 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - peering_event(id=649, detail=PeeringEvent(from=1 pgid=3.13 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - peering_event(id=645, detail=PeeringEvent(from=1 pgid=3.12 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - peering_event(id=644, detail=PeeringEvent(from=1 pgid=3.b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - peering_event(id=650, detail=PeeringEvent(from=1 pgid=3.15 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.066 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - peering_event(id=650, detail=PeeringEvent(from=1 pgid=3.15 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.f 2022-03-29T17:52:33.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.984573364s (sent 174.984573364s) 2022-03-29T17:52:33.067 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.068 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.984786987s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.069 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,288 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.070 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.071 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:33.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - peering_event(id=651, detail=PeeringEvent(from=1 pgid=3.f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.072 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.073 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:33.074 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - peering_event(id=638, detail=PeeringEvent(from=1 pgid=3.f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - do_recover_missing check for recovery, 3:535385f0:::rbd_children:head 2022-03-29T17:52:33.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_advance_map(id=223, detail=PGAdvanceMap(pg=1.0 from=19 to=20)): complete 2022-03-29T17:52:33.075 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_advance_map(id=254, detail=PGAdvanceMap(pg=3.1b from=19 to=20)): complete 2022-03-29T17:52:33.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - peering_event(id=649, detail=PeeringEvent(from=1 pgid=3.13 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - peering_event(id=646, detail=PeeringEvent(from=1 pgid=3.1f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - pg_advance_map(id=234, detail=PGAdvanceMap(pg=3.4 from=19 to=20)): complete 2022-03-29T17:52:33.076 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,289 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 1.0 2022-03-29T17:52:33.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded state: _activate_committed 20 peer_activated now 1 last_interval_started 8 last_epoch_started 9 same_interval_since 19 2022-03-29T17:52:33.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded state: all_activated_and_committed 2022-03-29T17:52:33.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded recalc_readable_until readable_until[_ub] 174.985977173s (sent 174.985977173s) 2022-03-29T17:52:33.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.077 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:33.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:33.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:33.078 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded update_calc_stats degraded 2 2022-03-29T17:52:33.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 activating+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=8/8 les/c/f=9/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded state: mnow 158.986206055s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded share_pg_info 2022-03-29T17:52:33.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.079 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:33.080 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:33.080 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.080 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:33.080 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 2 2022-03-29T17:52:33.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:21 2022-03-29T17:52:33.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded needs_recovery is recovered 2022-03-29T17:52:33.081 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded needs_backfill does not need backfill 2022-03-29T17:52:33.082 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.082 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - peering_event(id=652, detail=PeeringEvent(from=1 pgid=1.0 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.082 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.082 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:33.083 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:33.083 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.083 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:33.083 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 2 2022-03-29T17:52:33.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:22 2022-03-29T17:52:33.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.084 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - peering_event(id=642, detail=PeeringEvent(from=1 pgid=1.0 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,290 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.085 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.a 2022-03-29T17:52:33.085 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.056980 2 0.002791 2022-03-29T17:52:33.085 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,291 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.234049, 0.002791243 spent on 2 events 2022-03-29T17:52:33.086 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.086 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,291 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.086 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.086 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000058 0 0.000000 2022-03-29T17:52:33.086 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,291 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.2910795, 0.0 spent on 0 events 2022-03-29T17:52:33.087 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.087 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,291 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.087 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.087 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.087 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.088 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.088 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.088 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.088 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[3.a( v 18'3 (0'0,18'3] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized publish_stats_to_osd 20:8 2022-03-29T17:52:33.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.089 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - peering_event(id=636, detail=PeeringEvent(from=1 pgid=3.a sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 2.4 2022-03-29T17:52:33.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 11 last_epoch_started 12 same_interval_since 19 2022-03-29T17:52:33.090 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.987609863s (sent 174.987609863s) 2022-03-29T17:52:33.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.091 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.987808228s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.092 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,291 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.093 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.094 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:18 2022-03-29T17:52:33.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.095 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - peering_event(id=653, detail=PeeringEvent(from=1 pgid=2.4 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.096 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.097 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_epoch 20 pg[2.4( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:19 2022-03-29T17:52:33.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - peering_event(id=633, detail=PeeringEvent(from=1 pgid=2.4 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - client_request(id=168, detail=m=[osd_op(client.4179.0:90 3.a 3:535385f0:::rbd_children:head {call rbd.add_child in=59b} snapc 0={} RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e19) v8]): got obc lock 2022-03-29T17:52:33.098 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - with_head_obc 3:535385f0:::rbd_children:head 2022-03-29T17:52:33.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - with_head_obc: cache miss on 3:535385f0:::rbd_children:head 2022-03-29T17:52:33.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - pg_advance_map(id=253, detail=PGAdvanceMap(pg=3.1a from=19 to=20)): complete 2022-03-29T17:52:33.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,292 [shard 0] osd - peering_event(id=648, detail=PeeringEvent(from=1 pgid=2.5 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - peering_event(id=652, detail=PeeringEvent(from=1 pgid=1.0 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.099 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - peering_event(id=651, detail=PeeringEvent(from=1 pgid=3.f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_advance_map(id=235, detail=PGAdvanceMap(pg=3.5 from=19 to=20)): complete 2022-03-29T17:52:33.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - peering_event(id=653, detail=PeeringEvent(from=1 pgid=2.4 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_advance_map(id=248, detail=PGAdvanceMap(pg=3.14 from=19 to=20)): complete 2022-03-29T17:52:33.100 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - do_recover_missing check for recovery, 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:33.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_advance_map(id=247, detail=PGAdvanceMap(pg=3.13 from=19 to=20)): complete 2022-03-29T17:52:33.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_advance_map(id=249, detail=PGAdvanceMap(pg=3.15 from=19 to=20)): complete 2022-03-29T17:52:33.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - peering_event(id=654, detail=PeeringEvent(from=1 pgid=3.c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.15 2022-03-29T17:52:33.101 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.989776611s (sent 174.989776611s) 2022-03-29T17:52:33.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.102 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,293 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.103 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized state: mnow 158.989959717s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.104 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.105 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:33.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.106 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - peering_event(id=655, detail=PeeringEvent(from=1 pgid=3.15 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.107 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.108 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.15( v 18'5 (0'0,18'5] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'5 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:33.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - peering_event(id=650, detail=PeeringEvent(from=1 pgid=3.15 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.109 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.110 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.d 2022-03-29T17:52:33.110 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded exit Started/Primary/Active/Activating 0.044326 2 0.002622 2022-03-29T17:52:33.110 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,294 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2504075, 0.002622255 spent on 2 events 2022-03-29T17:52:33.110 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,294 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.111 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded needs_recovery is recovered 2022-03-29T17:52:33.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded exit Started/Primary/Active/Recovered 0.000052 0 0.000000 2022-03-29T17:52:33.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,294 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.2947757, 0.0 spent on 0 events 2022-03-29T17:52:33.111 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded enter Started/Primary/Active/Clean 2022-03-29T17:52:33.112 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,294 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded share_pg_info 2022-03-29T17:52:33.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.112 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats shard 1 primary objects 1 missing 0 2022-03-29T17:52:33.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {1=1} 2022-03-29T17:52:33.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,294 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 1 2022-03-29T17:52:33.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 1 2022-03-29T17:52:33.113 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.d( v 18'4 (0'0,18'4] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'4 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:17 2022-03-29T17:52:33.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - peering_event(id=637, detail=PeeringEvent(from=1 pgid=3.d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.114 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_advance_map(id=250, detail=PGAdvanceMap(pg=3.16 from=19 to=20)): complete 2022-03-29T17:52:33.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - peering_event(id=654, detail=PeeringEvent(from=1 pgid=3.c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.13 2022-03-29T17:52:33.115 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.991226196s (sent 174.991226196s) 2022-03-29T17:52:33.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.116 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.117 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.991424561s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.118 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.119 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:15 2022-03-29T17:52:33.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.120 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - peering_event(id=656, detail=PeeringEvent(from=1 pgid=3.13 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.121 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.122 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,295 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:16 2022-03-29T17:52:33.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - peering_event(id=649, detail=PeeringEvent(from=1 pgid=3.13 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.123 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.124 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_advance_map(id=256, detail=PGAdvanceMap(pg=3.1e from=19 to=20)): complete 2022-03-29T17:52:33.124 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - peering_event(id=657, detail=PeeringEvent(from=1 pgid=3.6 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.124 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 2.6 2022-03-29T17:52:33.124 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.054087 2 0.002548 2022-03-29T17:52:33.124 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2422078, 0.002548875 spent on 2 events 2022-03-29T17:52:33.125 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.125 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.125 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.125 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000051 0 0.000000 2022-03-29T17:52:33.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.2963433, 0.0 spent on 0 events 2022-03-29T17:52:33.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.126 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.126 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.127 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[2.6( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:20 2022-03-29T17:52:33.128 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - peering_event(id=643, detail=PeeringEvent(from=1 pgid=2.6 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.129 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.10 2022-03-29T17:52:33.129 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.038825 2 0.002145 2022-03-29T17:52:33.130 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2579148, 0.00214535 spent on 2 events 2022-03-29T17:52:33.130 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.130 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.130 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.130 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000057 0 0.000000 2022-03-29T17:52:33.131 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.2967818, 0.0 spent on 0 events 2022-03-29T17:52:33.131 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.131 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,296 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.131 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,296 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.132 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.10( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:17 2022-03-29T17:52:33.133 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - peering_event(id=641, detail=PeeringEvent(from=1 pgid=3.10 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - peering_event(id=658, detail=PeeringEvent(from=1 pgid=2.2 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.134 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.17 2022-03-29T17:52:33.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.993255615s (sent 174.993255615s) 2022-03-29T17:52:33.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.135 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.136 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized state: mnow 158.993453979s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.137 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.138 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:33.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.139 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - peering_event(id=659, detail=PeeringEvent(from=1 pgid=3.17 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.140 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.141 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,297 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.17( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:33.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - peering_event(id=640, detail=PeeringEvent(from=1 pgid=3.17 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.142 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - peering_event(id=657, detail=PeeringEvent(from=1 pgid=3.6 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - peering_event(id=660, detail=PeeringEvent(from=1 pgid=2.7 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): start 2022-03-29T17:52:33.143 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.1f 2022-03-29T17:52:33.143 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.050288 2 0.003046 2022-03-29T17:52:33.144 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,298 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2482698, 0.003046096 spent on 2 events 2022-03-29T17:52:33.144 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.144 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,298 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.144 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.144 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000048 0 0.000000 2022-03-29T17:52:33.145 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,298 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.298606, 0.0 spent on 0 events 2022-03-29T17:52:33.145 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.145 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,298 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.145 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.146 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.147 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - pg_epoch 20 pg[3.1f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:8 2022-03-29T17:52:33.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.148 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - peering_event(id=646, detail=PeeringEvent(from=1 pgid=3.1f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,298 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - peering_event(id=658, detail=PeeringEvent(from=1 pgid=2.2 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.1c 2022-03-29T17:52:33.149 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded state: all_activated_and_committed 2022-03-29T17:52:33.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded recalc_readable_until readable_until[_ub] 174.995086670s (sent 174.995086670s) 2022-03-29T17:52:33.150 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:33.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:33.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.151 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:33.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded update_calc_stats degraded 2 2022-03-29T17:52:33.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 activating+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded state: mnow 158.995285034s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.152 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded share_pg_info 2022-03-29T17:52:33.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:33.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:33.153 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:33.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 2 2022-03-29T17:52:33.154 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:15 2022-03-29T17:52:33.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded needs_recovery is recovered 2022-03-29T17:52:33.155 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded needs_backfill does not need backfill 2022-03-29T17:52:33.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - peering_event(id=661, detail=PeeringEvent(from=1 pgid=3.1c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.156 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:33.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:33.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:33.157 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 2 2022-03-29T17:52:33.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - pg_epoch 20 pg[3.1c( v 18'3 (0'0,18'3] local-lis/les=19/20 n=2 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=18'3 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:16 2022-03-29T17:52:33.158 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,299 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - peering_event(id=639, detail=PeeringEvent(from=1 pgid=3.1c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - peering_event(id=656, detail=PeeringEvent(from=1 pgid=3.13 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.159 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - peering_event(id=655, detail=PeeringEvent(from=1 pgid=3.15 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_advance_map(id=233, detail=PGAdvanceMap(pg=3.2 from=19 to=20)): complete 2022-03-29T17:52:33.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_advance_map(id=226, detail=PGAdvanceMap(pg=2.2 from=19 to=20)): complete 2022-03-29T17:52:33.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - peering_event(id=660, detail=PeeringEvent(from=1 pgid=2.7 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): pg present 2022-03-29T17:52:33.160 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.12 2022-03-29T17:52:33.161 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.061307 2 0.002402 2022-03-29T17:52:33.161 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,300 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2392082, 0.002402857 spent on 2 events 2022-03-29T17:52:33.161 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.161 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,300 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.162 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.162 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000058 0 0.000000 2022-03-29T17:52:33.162 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,300 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.3005633, 0.0 spent on 0 events 2022-03-29T17:52:33.162 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.163 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,300 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.163 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.164 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.165 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - pg_epoch 20 pg[3.12( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:8 2022-03-29T17:52:33.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - peering_event(id=645, detail=PeeringEvent(from=1 pgid=3.12 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.166 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,300 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.1b 2022-03-29T17:52:33.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 174.996963501s (sent 174.996963501s) 2022-03-29T17:52:33.167 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.168 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.169 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 158.997146606s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.170 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.171 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:33.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.172 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - peering_event(id=662, detail=PeeringEvent(from=1 pgid=3.1b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.173 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.174 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[3.1b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:33.175 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - peering_event(id=647, detail=PeeringEvent(from=1 pgid=3.1b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.176 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 2.5 2022-03-29T17:52:33.176 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.041475 2 0.003042 2022-03-29T17:52:33.177 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,301 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2604198, 0.003042924 spent on 2 events 2022-03-29T17:52:33.177 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.177 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,301 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.177 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,301 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.178 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000047 0 0.000000 2022-03-29T17:52:33.178 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.301937, 0.0 spent on 0 events 2022-03-29T17:52:33.178 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.178 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.179 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.180 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[2.5( empty local-lis/les=19/20 n=0 ec=11/11 lis/c=19/11 les/c/f=20/12/0 sis=19) [1] r=0 lpr=19 pi=[11,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:20 2022-03-29T17:52:33.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - peering_event(id=648, detail=PeeringEvent(from=1 pgid=2.5 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.181 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_advance_map(id=236, detail=PGAdvanceMap(pg=3.6 from=19 to=20)): complete 2022-03-29T17:52:33.182 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 1.0 2022-03-29T17:52:33.182 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded exit Started/Primary/Active/Activating 0.037861 2 0.002050 2022-03-29T17:52:33.182 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2646332, 0.002050792 spent on 2 events 2022-03-29T17:52:33.183 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.183 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.183 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded needs_recovery is recovered 2022-03-29T17:52:33.183 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded exit Started/Primary/Active/Recovered 0.000064 0 0.000000 2022-03-29T17:52:33.184 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.3025367, 0.0 spent on 0 events 2022-03-29T17:52:33.184 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded enter Started/Primary/Active/Clean 2022-03-29T17:52:33.184 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,302 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.184 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded share_pg_info 2022-03-29T17:52:33.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats shard 1 primary objects 2 missing 0 2022-03-29T17:52:33.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats object_location_counts {1=2} 2022-03-29T17:52:33.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.185 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats missing shard NONE missing= 2 2022-03-29T17:52:33.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats degraded 2 2022-03-29T17:52:33.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded update_calc_stats misplaced 0 2022-03-29T17:52:33.186 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - pg_epoch 20 pg[1.0( v 12'97 (0'0,12'97] local-lis/les=19/20 n=2 ec=8/8 lis/c=19/8 les/c/f=20/9/0 sis=19) [1] r=0 lpr=19 pi=[8,19)/1 crt=12'97 mlcod 0'0 active+undersized+degraded publish_stats_to_osd 20:23 2022-03-29T17:52:33.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - peering_event(id=652, detail=PeeringEvent(from=1 pgid=1.0 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.187 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,302 [shard 0] osd - do_recover_missing check for recovery, 3:a9e8038f:::rbd_header.1053953fed53:head 2022-03-29T17:52:33.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - peering_event(id=659, detail=PeeringEvent(from=1 pgid=3.17 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_advance_map(id=241, detail=PGAdvanceMap(pg=3.c from=19 to=20)): complete 2022-03-29T17:52:33.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_advance_map(id=224, detail=PGAdvanceMap(pg=2.0 from=19 to=20)): complete 2022-03-29T17:52:33.188 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.b 2022-03-29T17:52:33.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.071269 2 0.014743 2022-03-29T17:52:33.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2318778, 0.014743383 spent on 2 events 2022-03-29T17:52:33.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.189 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.190 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.190 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000068 0 0.000000 2022-03-29T17:52:33.190 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.3031886, 0.0 spent on 0 events 2022-03-29T17:52:33.190 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.191 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.191 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.192 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.b( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:17 2022-03-29T17:52:33.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.193 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - peering_event(id=644, detail=PeeringEvent(from=1 pgid=3.b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.194 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.f 2022-03-29T17:52:33.194 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.050983 2 0.002229 2022-03-29T17:52:33.195 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.2526026, 0.002229008 spent on 2 events 2022-03-29T17:52:33.195 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.195 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.195 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.195 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000044 0 0.000000 2022-03-29T17:52:33.196 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.3036258, 0.0 spent on 0 events 2022-03-29T17:52:33.196 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.196 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,303 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.196 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.197 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.198 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - pg_epoch 20 pg[3.f( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:8 2022-03-29T17:52:33.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - peering_event(id=651, detail=PeeringEvent(from=1 pgid=3.f sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.199 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,303 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,304 [shard 0] osd - load_metadata: object 3:535385f0:::rbd_children:head doesn't exist, returning empty metadata 2022-03-29T17:52:33.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,304 [shard 0] osd - load_head_obc: loaded obs 3:535385f0:::rbd_children:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:535385f0:::rbd_children:head 2022-03-29T17:52:33.200 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,304 [shard 0] osd - load_head_obc: returning obc 3:535385f0:::rbd_children:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:535385f0:::rbd_children:head 2022-03-29T17:52:33.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,304 [shard 0] osd - peering_event(id=661, detail=PeeringEvent(from=1 pgid=3.1c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,304 [shard 0] osd - peering_event(id=662, detail=PeeringEvent(from=1 pgid=3.1b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): pg present 2022-03-29T17:52:33.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,304 [shard 0] osd - pg_advance_map(id=231, detail=PGAdvanceMap(pg=2.7 from=19 to=20)): complete 2022-03-29T17:52:33.201 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,304 [shard 0] osd - pg_advance_map(id=227, detail=PGAdvanceMap(pg=2.3 from=19 to=20)): complete 2022-03-29T17:52:33.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,304 [shard 0] osd - client_request(id=170, detail=m=[osd_op(client.4179.0:106 3.a 3:5741e4f5:::rbd_header.1053eb73922:head {watch reconnect cookie 140032644115424 gen 2} snapc 0={} ondisk+write+known_if_redirected+supports_pool_eio e19) v8]): got obc lock 2022-03-29T17:52:33.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - with_head_obc 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:33.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - peering_event(id=631, detail=PeeringEvent(from=1 pgid=3.d sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:33.202 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - with_head_obc: cache miss on 3:5741e4f5:::rbd_header.1053eb73922:head 2022-03-29T17:52:33.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20 for pg: 3.c 2022-03-29T17:52:33.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: _activate_committed 20 peer_activated now 1 last_interval_started 14 last_epoch_started 15 same_interval_since 19 2022-03-29T17:52:33.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized state: all_activated_and_committed 2022-03-29T17:52:33.203 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized recalc_readable_until readable_until[_ub] 175.001129150s (sent 175.001129150s) 2022-03-29T17:52:33.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.204 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 activating+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.205 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized state: mnow 159.001327515s >= prior_readable_until_ub 0.000000000s 2022-03-29T17:52:33.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.206 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.207 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:6 2022-03-29T17:52:33.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.208 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_backfill does not need backfill 2022-03-29T17:52:33.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - for pg: {}: no need to recover or backfill, AllReplicasRecovered 2022-03-29T17:52:33.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - peering_event(id=663, detail=PeeringEvent(from=1 pgid=3.c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): start 2022-03-29T17:52:33.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.209 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.210 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.211 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - pg_epoch 20 pg[3.c( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:7 2022-03-29T17:52:33.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - peering_event(id=654, detail=PeeringEvent(from=1 pgid=3.c sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): submitting ctx 2022-03-29T17:52:33.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,305 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.212 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - peering_event(id=627, detail=PeeringEvent(from=1 pgid=3.a sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:33.213 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - do_peering_event handling epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered for pg: 3.13 2022-03-29T17:52:33.213 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Activating 0.032944 2 0.001434 2022-03-29T17:52:33.213 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,306 [shard 0] osd - Exiting state: Started/Primary/Active/Activating, entered at 1648576352.273352, 0.001434696 spent on 2 events 2022-03-29T17:52:33.214 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Recovered 2022-03-29T17:52:33.214 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,306 [shard 0] osd - Entering state: Started/Primary/Active/Recovered 2022-03-29T17:52:33.214 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized needs_recovery is recovered 2022-03-29T17:52:33.214 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized exit Started/Primary/Active/Recovered 0.000055 0 0.000000 2022-03-29T17:52:33.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,306 [shard 0] osd - Exiting state: Started/Primary/Active/Recovered, entered at 1648576352.3063414, 0.0 spent on 0 events 2022-03-29T17:52:33.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized enter Started/Primary/Active/Clean 2022-03-29T17:52:33.215 INFO:tasks.ceph.osd.1.smithi138.stderr:INFO 2022-03-29 17:52:32,306 [shard 0] osd - Entering state: Started/Primary/Active/Clean 2022-03-29T17:52:33.215 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized share_pg_info 2022-03-29T17:52:33.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats actingset 1 upset 1 acting_recovery_backfill 1 2022-03-29T17:52:33.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats shard 1 primary objects 0 missing 0 2022-03-29T17:52:33.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats object_location_counts {} 2022-03-29T17:52:33.216 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard 1 missing= 0 2022-03-29T17:52:33.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats missing shard NONE missing= 0 2022-03-29T17:52:33.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats degraded 0 2022-03-29T17:52:33.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized update_calc_stats misplaced 0 2022-03-29T17:52:33.217 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized prepare_stats_for_publish reporting purged_snaps [] 2022-03-29T17:52:33.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_epoch 20 pg[3.13( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [1] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized publish_stats_to_osd 20:17 2022-03-29T17:52:33.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - peering_event(id=656, detail=PeeringEvent(from=1 pgid=3.13 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 AllReplicasRecovered)): submitting ctx 2022-03-29T17:52:33.218 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,306 [shard 0] osd - pg_advance_map(id=238, detail=PGAdvanceMap(pg=3.8 from=19 to=20)): complete 2022-03-29T17:52:33.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,307 [shard 0] osd - peering_event(id=630, detail=PeeringEvent(from=1 pgid=3.12 sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:33.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,307 [shard 0] osd - peering_event(id=628, detail=PeeringEvent(from=1 pgid=3.b sent=20 requested=20 evt=epoch_sent: 20 epoch_requested: 20 ActivateCommitted from 20 processed at 20)): complete 2022-03-29T17:52:33.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,307 [shard 0] osd - do_osd_ops_execute: object 3:535385f0:::rbd_children:head - handling op call 2022-03-29T17:52:33.219 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,307 [shard 0] osd - handling op call on object 3:535385f0:::rbd_children:head 2022-03-29T17:52:33.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,307 [shard 0] osd - calling method rbd.add_child, num_read=0, num_write=0 2022-03-29T17:52:33.220 INFO:tasks.ceph.osd.1.smithi138.stderr:DEBUG 2022-03-29 17:52:32,307 [shard 0] osd - handling op omap-get-vals-by-keys on object 3:535385f0:::rbd_children:head 2022-03-29T17:52:33.220 INFO:tasks.ceph.osd.1.smithi138.stderr:================================================================= 2022-03-29T17:52:33.220 INFO:tasks.ceph.osd.1.smithi138.stderr:==35578==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fded8850308 at pc 0x562ab193d0f1 bp 0x7fded88502d0 sp 0x7fded88502c0 2022-03-29T17:52:33.221 INFO:tasks.ceph.osd.1.smithi138.stderr:READ of size 8 at 0x7fded8850308 thread T0 2022-03-29T17:52:33.221 INFO:tasks.ceph.osd.1.smithi138.stderr: #0 0x562ab193d0f0 (/usr/bin/ceph-osd+0x478d20f0) 2022-03-29T17:52:33.221 INFO:tasks.ceph.osd.1.smithi138.stderr: #1 0x562ab193c063 in dl_iterate_phdr (/usr/bin/ceph-osd+0x478d1063) 2022-03-29T17:52:33.221 INFO:tasks.ceph.osd.1.smithi138.stderr: #2 0x7fdef8aa5174 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13174) 2022-03-29T17:52:33.222 INFO:tasks.ceph.osd.1.smithi138.stderr: #3 0x7fdef8aa1712 (/lib64/libgcc_s.so.1+0xf712) 2022-03-29T17:52:33.222 INFO:tasks.ceph.osd.1.smithi138.stderr: #4 0x7fdef8aa28ef (/lib64/libgcc_s.so.1+0x108ef) 2022-03-29T17:52:33.222 INFO:tasks.ceph.osd.1.smithi138.stderr: #5 0x7fdef8aa332a in _Unwind_Resume (/lib64/libgcc_s.so.1+0x1132a) 2022-03-29T17:52:33.222 INFO:tasks.ceph.osd.1.smithi138.stderr: #6 0x562aa4bc7283 (/usr/bin/ceph-osd+0x3ab5c283) 2022-03-29T17:52:33.223 INFO:tasks.ceph.osd.1.smithi138.stderr: #7 0x562aa4bc8f33 in seastar::logger::lambda_log_writer(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x3ab5df33) 2022-03-29T17:52:33.223 INFO:tasks.ceph.osd.1.smithi138.stderr: #8 0x562ab2f68f81 (/usr/bin/ceph-osd+0x48efdf81) 2022-03-29T17:52:33.223 INFO:tasks.ceph.osd.1.smithi138.stderr: #9 0x562aa5fe94fa in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const (/usr/bin/ceph-osd+0x3bf7e4fa) 2022-03-29T17:52:33.223 INFO:tasks.ceph.osd.1.smithi138.stderr: #10 0x562aa625821c (/usr/bin/ceph-osd+0x3c1ed21c) 2022-03-29T17:52:33.223 INFO:tasks.ceph.osd.1.smithi138.stderr: #11 0x562aa63fe8b0 in crimson::osd::OpsExecuter::execute_op(OSDOp&) (/usr/bin/ceph-osd+0x3c3938b0) 2022-03-29T17:52:33.224 INFO:tasks.ceph.osd.1.smithi138.stderr: #12 0x562aa7409310 (/usr/bin/ceph-osd+0x3d39e310) 2022-03-29T17:52:33.224 INFO:tasks.ceph.osd.1.smithi138.stderr: #13 0x562aa746345b in cls_cxx_map_get_val(void*, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list*) (/usr/bin/ceph-osd+0x3d3f845b) 2022-03-29T17:52:33.224 INFO:tasks.ceph.osd.1.smithi138.stderr: #14 0x7fdee47945c0 (/usr/lib64/rados-classes/libcls_rbd.so.1.0.0+0x3e5c0) 2022-03-29T17:52:33.224 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:33.225 INFO:tasks.ceph.osd.1.smithi138.stderr:0x7fded8850308 is located 242440 bytes inside of 262144-byte region [0x7fded8815000,0x7fded8855000) 2022-03-29T17:52:33.225 INFO:tasks.ceph.osd.1.smithi138.stderr:allocated by thread T0 here: 2022-03-29T17:52:33.225 INFO:tasks.ceph.osd.1.smithi138.stderr: #0 0x7fdefcfb5780 in __interceptor_aligned_alloc (/lib64/libasan.so.5+0xf0780) 2022-03-29T17:52:33.225 INFO:tasks.ceph.osd.1.smithi138.stderr: #1 0x562ab22c81c1 (/usr/bin/ceph-osd+0x4825d1c1) 2022-03-29T17:52:33.226 INFO:tasks.ceph.osd.1.smithi138.stderr: #2 0x7ffd615785df ([stack]+0xa45df) 2022-03-29T17:52:33.226 INFO:tasks.ceph.osd.1.smithi138.stderr: 2022-03-29T17:52:33.226 INFO:tasks.ceph.osd.1.smithi138.stderr:SUMMARY: AddressSanitizer: stack-buffer-overflow (/usr/bin/ceph-osd+0x478d20f0) 2022-03-29T17:52:33.226 INFO:tasks.ceph.osd.1.smithi138.stderr:Shadow bytes around the buggy address: 2022-03-29T17:52:33.227 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b102010: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:33.227 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b102020: f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:33.227 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b102030: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:33.227 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b102040: f8 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 00 00 00 2022-03-29T17:52:33.227 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b102050: 00 00 00 00 00 00 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:33.228 INFO:tasks.ceph.osd.1.smithi138.stderr:=>0x0ffc5b102060: 00[f2]f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2022-03-29T17:52:33.228 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b102070: 00 f2 f2 f2 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 2022-03-29T17:52:33.228 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b102080: 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 2022-03-29T17:52:33.229 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b102090: 00 00 f2 f2 00 00 00 f2 f2 f2 f2 f2 00 00 00 00 2022-03-29T17:52:33.229 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b1020a0: f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00 00 00 2022-03-29T17:52:33.229 INFO:tasks.ceph.osd.1.smithi138.stderr: 0x0ffc5b1020b0: f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00 00 00 2022-03-29T17:52:33.229 INFO:tasks.ceph.osd.1.smithi138.stderr:Shadow byte legend (one shadow byte represents 8 application bytes): 2022-03-29T17:52:33.230 INFO:tasks.ceph.osd.1.smithi138.stderr: Addressable: 00 2022-03-29T17:52:33.230 INFO:tasks.ceph.osd.1.smithi138.stderr: Partially addressable: 01 02 03 04 05 06 07 2022-03-29T17:52:33.230 INFO:tasks.ceph.osd.1.smithi138.stderr: Heap left redzone: fa 2022-03-29T17:52:33.230 INFO:tasks.ceph.osd.1.smithi138.stderr: Freed heap region: fd 2022-03-29T17:52:33.231 INFO:tasks.ceph.osd.1.smithi138.stderr: Stack left redzone: f1 2022-03-29T17:52:33.231 INFO:tasks.ceph.osd.1.smithi138.stderr: Stack mid redzone: f2 2022-03-29T17:52:33.231 INFO:tasks.ceph.osd.1.smithi138.stderr: Stack right redzone: f3 2022-03-29T17:52:33.231 INFO:tasks.ceph.osd.1.smithi138.stderr: Stack after return: f5 2022-03-29T17:52:33.231 INFO:tasks.ceph.osd.1.smithi138.stderr: Stack use after scope: f8 2022-03-29T17:52:33.232 INFO:tasks.ceph.osd.1.smithi138.stderr: Global redzone: f9 2022-03-29T17:52:33.232 INFO:tasks.ceph.osd.1.smithi138.stderr: Global init order: f6 2022-03-29T17:52:33.232 INFO:tasks.ceph.osd.1.smithi138.stderr: Poisoned by user: f7 2022-03-29T17:52:33.232 INFO:tasks.ceph.osd.1.smithi138.stderr: Container overflow: fc 2022-03-29T17:52:33.233 INFO:tasks.ceph.osd.1.smithi138.stderr: Array cookie: ac 2022-03-29T17:52:33.233 INFO:tasks.ceph.osd.1.smithi138.stderr: Intra object redzone: bb 2022-03-29T17:52:33.233 INFO:tasks.ceph.osd.1.smithi138.stderr: ASan internal: fe 2022-03-29T17:52:33.233 INFO:tasks.ceph.osd.1.smithi138.stderr: Left alloca redzone: ca 2022-03-29T17:52:33.234 INFO:tasks.ceph.osd.1.smithi138.stderr: Right alloca redzone: cb 2022-03-29T17:52:33.234 INFO:tasks.ceph.osd.1.smithi138.stderr:==35578==ABORTING 2022-03-29T17:52:33.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,430 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579@53746 >> osd.1 v2:172.21.15.138:6807/35578] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:33.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,430 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579@53746 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset yes, replace no 2022-03-29T17:52:33.236 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,430 [shard 0] osd - Heartbeat::Peer: osd.1 disconnected 2022-03-29T17:52:33.237 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,430 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=99440837417440437 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:33.237 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,430 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578@58329] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:33.237 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,430 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578@58329] closing: reset yes, replace no 2022-03-29T17:52:33.237 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,430 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=877277914094303787 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:33.238 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,430 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@64300 >> osd.1 v2:172.21.15.138:6801/35578] execute_ready(): fault at READY with nothing to send, going to STANDBY -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:52:33.238 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.238 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:32,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:52:33.239 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.239 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:32,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:52:33.240 INFO:tasks.ceph.osd.1.smithi138.stderr:daemon-helper: command failed with exit status 1 2022-03-29T17:52:33.243 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,631 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:33.243 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,631 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:33.243 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,631 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.244 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:32,631 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:52:33.244 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,632 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.244 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:32,632 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:52:33.244 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,861 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:33.245 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,861 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:33.245 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.245 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:32,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:52:33.245 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:32,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:33.246 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,032 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:33.246 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,032 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:33.246 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,032 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.246 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:33,032 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:52:33.247 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,032 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.247 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:33,032 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:52:33.247 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,036 [shard 0] alienstore - stat 2022-03-29T17:52:33.834 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,832 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:33.834 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,833 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:33.835 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,833 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.836 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:33,833 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:52:33.836 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,833 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.837 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:33,833 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:52:33.873 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,872 [shard 0] osd - peering_event(id=340, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:33.875 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,872 [shard 0] osd - peering_event(id=340, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:33.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,873 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:52:33.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,873 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:52:33.876 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,873 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 176.506317139s) 2022-03-29T17:52:33.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@64300 >> osd.1 v2:172.21.15.138:6801/35578] notify_write(): at STANDBY, going to CONNECTING 2022-03-29T17:52:33.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579@64300 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:33.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,873 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,873 [shard 0] osd - peering_event(id=340, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:33.877 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,873 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:33.878 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:33,874 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 0.2 seconds ... 2022-03-29T17:52:33.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:33,874 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:33.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,874 [shard 0] osd - peering_event(id=340, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:33.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,882 [shard 0] osd - peering_event(id=341, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:33.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,882 [shard 0] osd - peering_event(id=341, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:33.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,882 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:52:33.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,882 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:52:33.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,882 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 176.515838623s) 2022-03-29T17:52:33.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,882 [shard 0] osd - log is not dirty 2022-03-29T17:52:33.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,882 [shard 0] osd - peering_event(id=341, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:33.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,882 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:33.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:33,883 [shard 0] osd - peering_event(id=341, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:34.045 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,043 [shard 0] alienstore - stat 2022-03-29T17:52:34.076 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,074 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:34.077 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,074 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:34.077 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,074 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:34.078 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:34,074 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 0.4 seconds ... 2022-03-29T17:52:34.078 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,074 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:34.477 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,474 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:34.477 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,475 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:34.477 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,475 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:34.478 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:34,475 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 0.8 seconds ... 2022-03-29T17:52:34.478 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:34,475 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:35.052 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,050 [shard 0] alienstore - stat 2022-03-29T17:52:35.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,275 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:35.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,275 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:35.278 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,275 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:35.278 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:35,275 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 1.6 seconds ... 2022-03-29T17:52:35.278 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,275 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:35.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,433 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:35.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,433 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:35.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,434 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:35.436 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:35,434 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:52:35.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:35,434 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:35.437 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:35,434 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:52:36.059 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:36,057 [shard 0] alienstore - stat 2022-03-29T17:52:36.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2022-03-29T17:52:36.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2022-03-29T17:52:36.361 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:36,359 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is pending send... 2022-03-29T17:52:36.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:36,876 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:36.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:36,876 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:36.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:36,876 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:36.879 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:36,876 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 3.2 seconds ... 2022-03-29T17:52:36.879 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:36,876 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:37.065 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:37,064 [shard 0] alienstore - stat 2022-03-29T17:52:38.072 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:38,071 [shard 0] alienstore - stat 2022-03-29T17:52:38.637 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:38,634 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:38.637 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:38,634 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:38.638 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:38,635 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:38.638 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:38,635 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 6.4 seconds ... 2022-03-29T17:52:38.638 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:38,635 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:38.639 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:38,635 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 6.4 seconds ... 2022-03-29T17:52:39.079 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:39,078 [shard 0] alienstore - stat 2022-03-29T17:52:39.492 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,490 [shard 0] osd - peering_event(id=342, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:39.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,490 [shard 0] osd - peering_event(id=342, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:39.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,490 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:52:39.493 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,490 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:52:39.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,490 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 182.123886108s) 2022-03-29T17:52:39.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,491 [shard 0] osd - log is not dirty 2022-03-29T17:52:39.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,491 [shard 0] osd - peering_event(id=342, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:39.494 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,491 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:39.495 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,491 [shard 0] osd - peering_event(id=342, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:39.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,500 [shard 0] osd - peering_event(id=343, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:39.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,500 [shard 0] osd - peering_event(id=343, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:39.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,500 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:52:39.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,500 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:52:39.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,500 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 182.133850098s) 2022-03-29T17:52:39.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,500 [shard 0] osd - log is not dirty 2022-03-29T17:52:39.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,500 [shard 0] osd - peering_event(id=343, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:39.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,500 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:39.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,501 [shard 0] osd - peering_event(id=343, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:39.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,530 [shard 0] osd - peering_event(id=344, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:39.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,530 [shard 0] osd - peering_event(id=344, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:39.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,531 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:52:39.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,531 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:52:39.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,531 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 182.164077759s) 2022-03-29T17:52:39.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,531 [shard 0] osd - log is not dirty 2022-03-29T17:52:39.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,531 [shard 0] osd - peering_event(id=344, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:39.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,531 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:39.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,531 [shard 0] osd - peering_event(id=344, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:39.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,561 [shard 0] osd - peering_event(id=345, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:39.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,561 [shard 0] osd - peering_event(id=345, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:39.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,561 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:52:39.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,561 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:52:39.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,561 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 182.194427490s) 2022-03-29T17:52:39.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,561 [shard 0] osd - log is not dirty 2022-03-29T17:52:39.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,561 [shard 0] osd - peering_event(id=345, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:39.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,561 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:39.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,562 [shard 0] osd - peering_event(id=345, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:39.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,571 [shard 0] osd - peering_event(id=346, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:39.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,571 [shard 0] osd - peering_event(id=346, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:39.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,571 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:52:39.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,571 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:52:39.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,571 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 182.204452515s) 2022-03-29T17:52:39.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,571 [shard 0] osd - log is not dirty 2022-03-29T17:52:39.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,571 [shard 0] osd - peering_event(id=346, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:39.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,571 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:39.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,572 [shard 0] osd - peering_event(id=346, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:39.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,601 [shard 0] osd - peering_event(id=347, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:39.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,601 [shard 0] osd - peering_event(id=347, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:39.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,601 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:52:39.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,601 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:52:39.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,601 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 182.234634399s) 2022-03-29T17:52:39.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,601 [shard 0] osd - log is not dirty 2022-03-29T17:52:39.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,601 [shard 0] osd - peering_event(id=347, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:39.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,601 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:39.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:39,602 [shard 0] osd - peering_event(id=347, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:40.078 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:40,076 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:40.079 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:40,077 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:40.079 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:40,077 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:40.079 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:40,077 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 6.4 seconds ... 2022-03-29T17:52:40.079 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:40,077 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:40.086 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:40,085 [shard 0] alienstore - stat 2022-03-29T17:52:40.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,235 [shard 0] osd - peering_event(id=348, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:40.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,235 [shard 0] osd - peering_event(id=349, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:40.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,235 [shard 0] osd - peering_event(id=350, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:40.240 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,235 [shard 0] osd - peering_event(id=348, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:40.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,235 [shard 0] osd - peering_event(id=349, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:40.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,235 [shard 0] osd - peering_event(id=350, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:40.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:52:40.241 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 182.869216919s (sent 182.869216919s) 2022-03-29T17:52:40.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - log is not dirty 2022-03-29T17:52:40.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - peering_event(id=350, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:40.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:40.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:52:40.242 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 182.869461060s (sent 182.869461060s) 2022-03-29T17:52:40.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - log is not dirty 2022-03-29T17:52:40.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - peering_event(id=349, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:40.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:40.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:52:40.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 182.869644165s (sent 182.869644165s) 2022-03-29T17:52:40.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - log is not dirty 2022-03-29T17:52:40.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - peering_event(id=348, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:40.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,236 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:40.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,237 [shard 0] osd - peering_event(id=349, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:40.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,237 [shard 0] osd - peering_event(id=350, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:40.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,237 [shard 0] osd - peering_event(id=348, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:40.259 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,255 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:40.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,255 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:40.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,255 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:40.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,255 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:40.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,255 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:40.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:40.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:52:40.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 182.889205933s (sent 182.889205933s) 2022-03-29T17:52:40.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - log is not dirty 2022-03-29T17:52:40.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:40.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:40.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:52:40.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 182.889419556s (sent 182.889419556s) 2022-03-29T17:52:40.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - log is not dirty 2022-03-29T17:52:40.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:40.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:40.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:52:40.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 182.889633179s (sent 182.889633179s) 2022-03-29T17:52:40.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - log is not dirty 2022-03-29T17:52:40.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:40.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,256 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:40.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,257 [shard 0] osd - peering_event(id=351, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:40.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,257 [shard 0] osd - peering_event(id=353, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:40.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:40,257 [shard 0] osd - peering_event(id=352, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:41.094 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:41,091 [shard 0] alienstore - stat 2022-03-29T17:52:41.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2022-03-29T17:52:41.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2022-03-29T17:52:41.879 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,877 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:41.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,877 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:41.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,877 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:52:41.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,877 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:52:41.880 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,877 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 184.510528564s) 2022-03-29T17:52:41.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,877 [shard 0] osd - log is not dirty 2022-03-29T17:52:41.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,877 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:41.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,877 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:41.881 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,878 [shard 0] osd - peering_event(id=354, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:41.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,887 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:41.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,887 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:41.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,887 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:52:41.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,887 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:52:41.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,887 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 184.520492554s) 2022-03-29T17:52:41.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,887 [shard 0] osd - log is not dirty 2022-03-29T17:52:41.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,887 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:41.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,887 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:41.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:41,888 [shard 0] osd - peering_event(id=355, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:42.100 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,098 [shard 0] alienstore - stat 2022-03-29T17:52:42.363 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:42,360 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:52:42.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,360 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:52:42.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,360 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=16014009973262802232 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:42.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,360 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:52:42.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,360 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=4245450709177776355 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:42.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,360 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:42.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,361 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:42.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,361 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:42.366 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:42,361 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:52:42.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,361 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:42.367 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:42,361 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:52:42.563 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,561 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:42.564 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,561 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:42.564 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,562 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:42.564 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:42,562 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:52:42.564 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,562 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:42.565 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:42,562 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:52:42.964 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,962 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:42.964 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,962 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:42.965 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,963 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:42.965 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:42,963 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:52:42.965 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:42,963 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:42.965 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:42,963 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:52:43.107 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:43,105 [shard 0] alienstore - stat 2022-03-29T17:52:43.765 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:43,763 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:43.765 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:43,763 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:43.766 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:43,763 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:43.766 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:43,763 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:52:43.766 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:43,763 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:43.766 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:43,764 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:52:44.113 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:44,112 [shard 0] alienstore - stat 2022-03-29T17:52:45.120 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:45,119 [shard 0] alienstore - stat 2022-03-29T17:52:45.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:45,364 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:45.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:45,364 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:45.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:45,364 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:45.367 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:45,364 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:52:45.367 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:45,364 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:45.367 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:45,364 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:52:46.127 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:46,126 [shard 0] alienstore - stat 2022-03-29T17:52:46.479 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:46,477 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:46.480 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:46,477 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:46.480 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:46,478 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:46.480 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:46,478 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 12.8 seconds ... 2022-03-29T17:52:46.480 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:46,478 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:46.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~38s 2022-03-29T17:52:46.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2022-03-29T17:52:47.134 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:47,132 [shard 0] alienstore - stat 2022-03-29T17:52:47.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,494 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:47.497 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,495 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:47.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,495 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:52:47.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,495 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:52:47.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,495 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 190.128372192s) 2022-03-29T17:52:47.498 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,495 [shard 0] osd - log is not dirty 2022-03-29T17:52:47.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,495 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:47.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,495 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:47.499 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,496 [shard 0] osd - peering_event(id=356, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:47.507 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - peering_event(id=357, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:47.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - peering_event(id=357, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:47.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:52:47.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:52:47.508 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 190.138214111s) 2022-03-29T17:52:47.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - log is not dirty 2022-03-29T17:52:47.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - peering_event(id=357, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:47.509 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:47.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,505 [shard 0] osd - peering_event(id=357, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:47.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,535 [shard 0] osd - peering_event(id=358, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:47.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,535 [shard 0] osd - peering_event(id=358, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:47.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,535 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:52:47.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,535 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:52:47.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,535 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 190.168563843s) 2022-03-29T17:52:47.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,535 [shard 0] osd - log is not dirty 2022-03-29T17:52:47.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,535 [shard 0] osd - peering_event(id=358, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:47.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,535 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:47.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,536 [shard 0] osd - peering_event(id=358, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:47.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,565 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:47.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,565 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:47.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,565 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:52:47.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,565 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:52:47.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,565 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 190.198806763s) 2022-03-29T17:52:47.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,565 [shard 0] osd - log is not dirty 2022-03-29T17:52:47.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,565 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:47.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,565 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:47.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,566 [shard 0] osd - peering_event(id=359, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:47.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,575 [shard 0] osd - peering_event(id=360, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:47.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,575 [shard 0] osd - peering_event(id=360, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:47.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,575 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:52:47.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,575 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:52:47.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,575 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 190.208862305s) 2022-03-29T17:52:47.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,575 [shard 0] osd - log is not dirty 2022-03-29T17:52:47.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,575 [shard 0] osd - peering_event(id=360, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:47.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,575 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:47.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,576 [shard 0] osd - peering_event(id=360, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:47.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,605 [shard 0] osd - peering_event(id=361, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:47.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,605 [shard 0] osd - peering_event(id=361, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:47.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,605 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:52:47.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,605 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:52:47.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,606 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 190.238983154s) 2022-03-29T17:52:47.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,606 [shard 0] osd - log is not dirty 2022-03-29T17:52:47.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,606 [shard 0] osd - peering_event(id=361, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:47.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,606 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:47.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:47,606 [shard 0] osd - peering_event(id=361, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:47.864 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:47,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:52:47.864 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:47,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:47.864 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:47,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:47.865 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:47,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:52:47.865 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:47,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:52:48.141 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,139 [shard 0] alienstore - stat 2022-03-29T17:52:48.243 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:48.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - peering_event(id=363, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:48.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - peering_event(id=364, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:48.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:48.244 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - peering_event(id=363, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:48.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - peering_event(id=364, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:48.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:52:48.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 190.873764038s (sent 190.873764038s) 2022-03-29T17:52:48.245 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - log is not dirty 2022-03-29T17:52:48.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:48.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:48.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,240 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:52:48.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 190.874023438s (sent 190.874023438s) 2022-03-29T17:52:48.246 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - log is not dirty 2022-03-29T17:52:48.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - peering_event(id=363, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:48.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:48.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:52:48.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 190.874206543s (sent 190.874206543s) 2022-03-29T17:52:48.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - log is not dirty 2022-03-29T17:52:48.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - peering_event(id=364, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:48.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:48.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - peering_event(id=362, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:48.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - peering_event(id=363, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:48.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,241 [shard 0] osd - peering_event(id=364, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:48.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:48.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - peering_event(id=366, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:48.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:48.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:48.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:48.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - peering_event(id=366, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:48.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:52:48.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 190.893722534s (sent 190.893722534s) 2022-03-29T17:52:48.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - log is not dirty 2022-03-29T17:52:48.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:48.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:48.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:52:48.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 190.893951416s (sent 190.893951416s) 2022-03-29T17:52:48.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - log is not dirty 2022-03-29T17:52:48.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:48.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,260 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:48.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,261 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:52:48.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,261 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 190.894134521s (sent 190.894134521s) 2022-03-29T17:52:48.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,261 [shard 0] osd - log is not dirty 2022-03-29T17:52:48.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,261 [shard 0] osd - peering_event(id=366, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:48.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,261 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:48.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,261 [shard 0] osd - peering_event(id=367, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:48.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,262 [shard 0] osd - peering_event(id=366, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:48.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:48,262 [shard 0] osd - peering_event(id=365, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:48.363 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:48,361 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:52:48.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,361 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:52:48.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,361 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=17702483989110335993 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:48.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,361 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:52:48.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,361 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=16804985955932014560 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:48.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,361 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:48.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,361 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:48.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,362 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:48.365 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:48,362 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:52:48.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,362 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:48.366 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:48,362 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:52:48.564 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,562 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:48.564 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,562 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:48.565 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,563 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:48.565 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:48,563 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:52:48.565 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,563 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:48.565 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:48,563 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:52:48.966 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,963 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:48.966 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,963 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:48.966 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,963 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:48.967 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:48,963 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:52:48.967 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:48,963 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:48.967 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:48,964 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:52:49.150 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:49,146 [shard 0] alienstore - stat 2022-03-29T17:52:49.767 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:49,764 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:49.767 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:49,764 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:49.768 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:49,764 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:49.768 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:49,764 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:52:49.768 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:49,764 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:49.769 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:49,764 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:52:49.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,881 [shard 0] osd - peering_event(id=368, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:49.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,881 [shard 0] osd - peering_event(id=368, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:49.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,881 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:52:49.884 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,881 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:52:49.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,881 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 192.514739990s) 2022-03-29T17:52:49.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,881 [shard 0] osd - log is not dirty 2022-03-29T17:52:49.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,881 [shard 0] osd - peering_event(id=368, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:49.885 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,881 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:49.886 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,882 [shard 0] osd - peering_event(id=368, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:49.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,891 [shard 0] osd - peering_event(id=369, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:49.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,891 [shard 0] osd - peering_event(id=369, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:49.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,891 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:52:49.894 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,891 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:52:49.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,891 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 192.524703979s) 2022-03-29T17:52:49.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,891 [shard 0] osd - log is not dirty 2022-03-29T17:52:49.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,891 [shard 0] osd - peering_event(id=369, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:49.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,891 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:49.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:49,892 [shard 0] osd - peering_event(id=369, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:50.155 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:50,153 [shard 0] alienstore - stat 2022-03-29T17:52:51.161 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:51,160 [shard 0] alienstore - stat 2022-03-29T17:52:51.367 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:51,364 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:51.368 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:51,364 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:51.368 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:51,365 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:51.368 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:51,365 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:52:51.368 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:51,365 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:51.369 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:51,365 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:52:52.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~43s 2022-03-29T17:52:52.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2022-03-29T17:52:52.168 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:52,167 [shard 0] alienstore - stat 2022-03-29T17:52:53.175 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:53,174 [shard 0] alienstore - stat 2022-03-29T17:52:54.021 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,019 [shard 0] monc - sending auth(proto 2 165 bytes epoch 0) v1 2022-03-29T17:52:54.021 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,019 [shard 0] monc - waiting 2022-03-29T17:52:54.022 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,020 [shard 0] monc - handle_auth_reply mon v2:172.21.15.138:6803/35579 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:52:54.022 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,020 [shard 0] monc - handle_auth_reply 2022-03-29T17:52:54.023 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,020 [shard 0] monc - do_auth_single: mon v2:172.21.15.138:6803/35579 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:52:54.182 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,180 [shard 0] alienstore - stat 2022-03-29T17:52:54.364 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:54,361 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:52:54.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,361 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:52:54.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,361 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=1542845221871764649 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:54.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,361 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:52:54.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,361 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=17921591338612906423 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:52:54.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,362 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:54.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,362 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:52:54.366 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,362 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:54.366 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:54,362 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:52:54.367 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,362 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:54.367 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:54,362 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:52:54.564 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,562 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:54.564 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,563 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:54.565 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,563 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:54.566 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:54,563 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:52:54.566 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,563 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:54.566 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:54,563 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:52:54.967 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,963 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:54.967 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,963 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:54.968 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,964 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:54.968 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:54,964 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:52:54.968 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:54,964 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:54.969 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:54,964 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:52:55.191 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:55,187 [shard 0] alienstore - stat 2022-03-29T17:52:55.501 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,498 [shard 0] osd - peering_event(id=370, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:55.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,499 [shard 0] osd - peering_event(id=370, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:55.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,499 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:52:55.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,499 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:52:55.502 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,499 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 198.132247925s) 2022-03-29T17:52:55.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,499 [shard 0] osd - log is not dirty 2022-03-29T17:52:55.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,499 [shard 0] osd - peering_event(id=370, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:55.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,499 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:55.503 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,500 [shard 0] osd - peering_event(id=370, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:55.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,509 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:55.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,509 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:55.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,509 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:52:55.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,509 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:52:55.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,509 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 198.142318726s) 2022-03-29T17:52:55.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,509 [shard 0] osd - log is not dirty 2022-03-29T17:52:55.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,509 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:55.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,509 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:55.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,510 [shard 0] osd - peering_event(id=371, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:55.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,539 [shard 0] osd - peering_event(id=372, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:55.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,539 [shard 0] osd - peering_event(id=372, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:55.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,539 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:52:55.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,539 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:52:55.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,539 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 198.172668457s) 2022-03-29T17:52:55.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,539 [shard 0] osd - log is not dirty 2022-03-29T17:52:55.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,539 [shard 0] osd - peering_event(id=372, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:55.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,539 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:55.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,540 [shard 0] osd - peering_event(id=372, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:55.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,569 [shard 0] osd - peering_event(id=373, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:55.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,569 [shard 0] osd - peering_event(id=373, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:55.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,569 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:52:55.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,569 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:52:55.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,569 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 198.202682495s) 2022-03-29T17:52:55.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,569 [shard 0] osd - log is not dirty 2022-03-29T17:52:55.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,569 [shard 0] osd - peering_event(id=373, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:55.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,569 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:55.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,570 [shard 0] osd - peering_event(id=373, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:55.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,579 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:55.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,579 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:55.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,579 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:52:55.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,579 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:52:55.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,579 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 198.212768555s) 2022-03-29T17:52:55.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,579 [shard 0] osd - log is not dirty 2022-03-29T17:52:55.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,579 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:55.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,579 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:55.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,580 [shard 0] osd - peering_event(id=374, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:55.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,609 [shard 0] osd - peering_event(id=375, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:52:55.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,609 [shard 0] osd - peering_event(id=375, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:52:55.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,609 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:52:55.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,609 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:52:55.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,609 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 198.242904663s) 2022-03-29T17:52:55.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,609 [shard 0] osd - log is not dirty 2022-03-29T17:52:55.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,609 [shard 0] osd - peering_event(id=375, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:52:55.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,610 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:55.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:55,610 [shard 0] osd - peering_event(id=375, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:52:55.766 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:55,764 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:55.766 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:55,764 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:55.767 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:55,765 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:55.767 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:55,765 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:52:55.768 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:55,765 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:55.768 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:55,765 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:52:56.195 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:56,193 [shard 0] alienstore - stat 2022-03-29T17:52:56.247 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,243 [shard 0] osd - peering_event(id=376, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:56.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,243 [shard 0] osd - peering_event(id=377, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:56.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,243 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:56.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,243 [shard 0] osd - peering_event(id=376, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:56.248 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:56.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - peering_event(id=377, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:56.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:52:56.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 198.877349854s (sent 198.877349854s) 2022-03-29T17:52:56.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - log is not dirty 2022-03-29T17:52:56.249 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - peering_event(id=377, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:56.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:56.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:52:56.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 198.877685547s (sent 198.877685547s) 2022-03-29T17:52:56.250 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - log is not dirty 2022-03-29T17:52:56.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:56.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:56.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:52:56.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 198.877944946s (sent 198.877944946s) 2022-03-29T17:52:56.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - log is not dirty 2022-03-29T17:52:56.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - peering_event(id=376, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:56.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,244 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:56.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,245 [shard 0] osd - peering_event(id=377, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:56.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,245 [shard 0] osd - peering_event(id=378, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:56.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,245 [shard 0] osd - peering_event(id=376, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:56.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,263 [shard 0] osd - peering_event(id=379, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:56.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - peering_event(id=380, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:56.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:52:56.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - peering_event(id=380, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:56.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:56.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - peering_event(id=379, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:52:56.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:52:56.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 198.897598267s (sent 198.897598267s) 2022-03-29T17:52:56.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - log is not dirty 2022-03-29T17:52:56.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - peering_event(id=380, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:56.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:56.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:52:56.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 198.897827148s (sent 198.897827148s) 2022-03-29T17:52:56.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - log is not dirty 2022-03-29T17:52:56.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:56.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:56.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,264 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:52:56.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,265 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 198.898025513s (sent 198.898025513s) 2022-03-29T17:52:56.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,265 [shard 0] osd - log is not dirty 2022-03-29T17:52:56.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,265 [shard 0] osd - peering_event(id=379, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:52:56.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,265 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:56.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,265 [shard 0] osd - peering_event(id=380, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:56.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,268 [shard 0] osd - peering_event(id=381, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:56.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,268 [shard 0] osd - peering_event(id=379, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:52:56.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,745 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4127.0:5 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:56.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,745 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4127.0:5 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:56.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,745 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4127.0:5 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 11 2022-03-29T17:52:56.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,745 [shard 0] osd - OpSequencer::start_op: op=7, last_started=2, last_unblocked=2, last_completed=2 2022-03-29T17:52:56.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,745 [shard 0] osd - do_recover_missing check for recovery, 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,746 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4127.0:5 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:56.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,746 [shard 0] osd - with_head_obc 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,746 [shard 0] osd - with_head_obc: found 2:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:52:56.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,746 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:52:56.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,746 [shard 0] osd - handling op omap-get-vals on object 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.758 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,746 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:52:56.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,746 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:5 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 2:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:52:56.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,746 [shard 0] osd - with_head_obc: released 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,747 [shard 0] osd - OpSequencer::finish_op_in_order: op=7, last_started=7, last_unblocked=7, last_completed=2 2022-03-29T17:52:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,747 [shard 0] osd - client_request(id=7, detail=m=[osd_op(client.4127.0:5 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,747 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4127.0:6 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): start 2022-03-29T17:52:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,747 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4127.0:6 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): in repeat 2022-03-29T17:52:56.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,747 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4127.0:6 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e18) v8]) same_interval_since: 14 2022-03-29T17:52:56.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - OpSequencer::start_op: op=8, last_started=0, last_unblocked=0, last_completed=0 2022-03-29T17:52:56.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - do_recover_missing check for recovery, 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4127.0:6 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): got obc lock 2022-03-29T17:52:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - with_head_obc 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - with_head_obc: cache miss on 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - load_metadata: object 3:88c1567c:::rbd_trash_purge_schedule:head doesn't exist, returning empty metadata 2022-03-29T17:52:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - load_head_obc: loaded obs 3:88c1567c:::rbd_trash_purge_schedule:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - load_head_obc: returning obc 3:88c1567c:::rbd_trash_purge_schedule:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) for 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:52:56.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - handling op omap-get-vals on object 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:52:56.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,748 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:6 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8 - object 3:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:52:56.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,749 [shard 0] osd - with_head_obc: released 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:52:56.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,749 [shard 0] osd - OpSequencer::finish_op_in_order: op=8, last_started=8, last_unblocked=8, last_completed=0 2022-03-29T17:52:56.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:56,749 [shard 0] osd - client_request(id=8, detail=m=[osd_op(client.4127.0:6 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e18) v8]): destroying 2022-03-29T17:52:57.201 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:57,200 [shard 0] alienstore - stat 2022-03-29T17:52:57.367 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:57,365 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:57.368 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:57,365 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:57.368 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:57,366 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:57.368 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:57,366 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:52:57.368 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:57,366 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:57.369 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:57,366 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:52:57.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2022-03-29T17:52:57.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~21s 2022-03-29T17:52:57.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,884 [shard 0] osd - peering_event(id=382, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:57.887 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,884 [shard 0] osd - peering_event(id=382, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:57.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,884 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:52:57.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,884 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:52:57.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,885 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 200.517929077s) 2022-03-29T17:52:57.888 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,885 [shard 0] osd - log is not dirty 2022-03-29T17:52:57.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,885 [shard 0] osd - peering_event(id=382, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:57.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,885 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:57.889 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,886 [shard 0] osd - peering_event(id=382, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:57.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,894 [shard 0] osd - peering_event(id=383, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:52:57.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,894 [shard 0] osd - peering_event(id=383, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:52:57.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,894 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:52:57.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,894 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:52:57.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,894 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 200.527786255s) 2022-03-29T17:52:57.898 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,894 [shard 0] osd - log is not dirty 2022-03-29T17:52:57.899 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,894 [shard 0] osd - peering_event(id=383, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:52:57.899 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,894 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:52:57.899 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:52:57,895 [shard 0] osd - peering_event(id=383, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:52:58.208 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:58,207 [shard 0] alienstore - stat 2022-03-29T17:52:59.216 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:59,213 [shard 0] alienstore - stat 2022-03-29T17:52:59.280 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:59,278 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:52:59.280 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:59,278 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:52:59.280 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:59,278 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:52:59.281 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:52:59,278 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:52:59.281 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:52:59,278 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:00.222 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,220 [shard 0] alienstore - stat 2022-03-29T17:53:00.373 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:00,371 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:00.374 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,371 [shard 0] osd - add_pending: osd.1 failed for 30 2022-03-29T17:53:00.374 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:00,371 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:00.374 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,371 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:00.374 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,371 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=13180907933663951262 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:00.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,371 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:00.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,371 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=17212288255652004807 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:00.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,371 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:00.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,372 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:00.376 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,372 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:00.376 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:00,372 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:00.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,372 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:00.377 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:00,372 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:00.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,573 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:00.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,573 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,573 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:00,573 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,573 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:00.576 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:00,573 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:00.977 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,973 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:00.977 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,974 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:00.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,974 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:00.979 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:00,974 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:00.979 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:00,974 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:00.979 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:00,974 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:01.228 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:01,226 [shard 0] alienstore - stat 2022-03-29T17:53:01.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:01,774 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:01.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:01,774 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:01.777 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:01,775 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:01.778 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:01,775 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:01.778 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:01,775 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:01.778 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:01,775 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:02.235 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:02,233 [shard 0] alienstore - stat 2022-03-29T17:53:02.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~54s 2022-03-29T17:53:02.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2022-03-29T17:53:02.865 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:02,862 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:53:02.865 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:02,863 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:53:02.865 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:02,863 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:02.866 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:02,863 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:53:02.866 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:02,863 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:53:03.242 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:03,240 [shard 0] alienstore - stat 2022-03-29T17:53:03.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:03,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:03.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:03,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:03.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:03,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:03.378 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:03,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:03.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:03,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:03.379 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:03,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:03.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,502 [shard 0] osd - peering_event(id=384, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:03.504 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,502 [shard 0] osd - peering_event(id=384, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:03.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,502 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:53:03.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,502 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:53:03.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,502 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 206.135574341s) 2022-03-29T17:53:03.505 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,502 [shard 0] osd - log is not dirty 2022-03-29T17:53:03.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,502 [shard 0] osd - peering_event(id=384, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:03.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,502 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:03.506 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,503 [shard 0] osd - peering_event(id=384, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,512 [shard 0] osd - peering_event(id=385, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,512 [shard 0] osd - peering_event(id=385, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,512 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:53:03.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,512 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:53:03.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,512 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 206.145385742s) 2022-03-29T17:53:03.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,512 [shard 0] osd - log is not dirty 2022-03-29T17:53:03.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,512 [shard 0] osd - peering_event(id=385, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:03.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,512 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:03.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,513 [shard 0] osd - peering_event(id=385, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:03.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,542 [shard 0] osd - peering_event(id=386, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:03.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,542 [shard 0] osd - peering_event(id=386, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:03.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,542 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:53:03.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,542 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:53:03.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,542 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 206.175811768s) 2022-03-29T17:53:03.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,542 [shard 0] osd - log is not dirty 2022-03-29T17:53:03.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,542 [shard 0] osd - peering_event(id=386, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:03.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,542 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:03.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,543 [shard 0] osd - peering_event(id=386, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:03.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,572 [shard 0] osd - peering_event(id=387, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:03.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,572 [shard 0] osd - peering_event(id=387, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:03.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,572 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:53:03.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,572 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:53:03.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,572 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 206.205825806s) 2022-03-29T17:53:03.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,572 [shard 0] osd - log is not dirty 2022-03-29T17:53:03.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,572 [shard 0] osd - peering_event(id=387, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:03.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,572 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:03.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,573 [shard 0] osd - peering_event(id=387, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:03.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,582 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:03.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,582 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:03.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,582 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:53:03.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,583 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:53:03.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,583 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 206.216033936s) 2022-03-29T17:53:03.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,583 [shard 0] osd - log is not dirty 2022-03-29T17:53:03.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,583 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:03.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,583 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:03.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,583 [shard 0] osd - peering_event(id=388, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:03.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,612 [shard 0] osd - peering_event(id=389, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:03.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,612 [shard 0] osd - peering_event(id=389, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:03.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,612 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:53:03.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,613 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:53:03.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,613 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 206.246047974s) 2022-03-29T17:53:03.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,613 [shard 0] osd - log is not dirty 2022-03-29T17:53:03.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,613 [shard 0] osd - peering_event(id=389, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:03.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,613 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:03.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:03,613 [shard 0] osd - peering_event(id=389, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:04.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,247 [shard 0] osd - peering_event(id=390, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:04.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,247 [shard 0] osd - peering_event(id=391, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:04.251 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,247 [shard 0] osd - peering_event(id=392, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:04.252 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:04,247 [shard 0] alienstore - stat 2022-03-29T17:53:04.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,247 [shard 0] osd - peering_event(id=390, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:04.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,247 [shard 0] osd - peering_event(id=392, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:04.252 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - peering_event(id=391, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:04.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:53:04.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 206.881240845s (sent 206.881240845s) 2022-03-29T17:53:04.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - log is not dirty 2022-03-29T17:53:04.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - peering_event(id=391, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:04.253 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:04.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:53:04.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 206.881530762s (sent 206.881530762s) 2022-03-29T17:53:04.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - log is not dirty 2022-03-29T17:53:04.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - peering_event(id=390, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:04.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:04.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:53:04.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 206.881774902s (sent 206.881774902s) 2022-03-29T17:53:04.255 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - log is not dirty 2022-03-29T17:53:04.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - peering_event(id=392, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:04.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,248 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:04.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,249 [shard 0] osd - peering_event(id=391, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:04.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,249 [shard 0] osd - peering_event(id=392, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:04.257 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,249 [shard 0] osd - peering_event(id=390, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:04.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:04.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - peering_event(id=394, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:04.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:04.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:04.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - peering_event(id=394, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:04.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:04.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:53:04.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 206.900924683s (sent 206.900924683s) 2022-03-29T17:53:04.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - log is not dirty 2022-03-29T17:53:04.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,267 [shard 0] osd - peering_event(id=394, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:04.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:04.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:53:04.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 206.901199341s (sent 206.901199341s) 2022-03-29T17:53:04.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - log is not dirty 2022-03-29T17:53:04.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:04.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:04.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:53:04.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 206.901351929s (sent 206.901351929s) 2022-03-29T17:53:04.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - log is not dirty 2022-03-29T17:53:04.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:04.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:04.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - peering_event(id=394, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:04.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,268 [shard 0] osd - peering_event(id=393, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:04.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:04,269 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:05.255 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:05,254 [shard 0] alienstore - stat 2022-03-29T17:53:05.890 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,888 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:05.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,888 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:05.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,888 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:53:05.891 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,888 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:53:05.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,888 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 208.521835327s) 2022-03-29T17:53:05.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,888 [shard 0] osd - log is not dirty 2022-03-29T17:53:05.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,889 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:05.892 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,889 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:05.893 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,889 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:05.900 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,898 [shard 0] osd - peering_event(id=397, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:05.900 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,898 [shard 0] osd - peering_event(id=397, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:05.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,898 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:53:05.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,898 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:53:05.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,898 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 208.531723022s) 2022-03-29T17:53:05.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,898 [shard 0] osd - log is not dirty 2022-03-29T17:53:05.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,898 [shard 0] osd - peering_event(id=397, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:05.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,898 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:05.902 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:05,899 [shard 0] osd - peering_event(id=397, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:06.262 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,261 [shard 0] alienstore - stat 2022-03-29T17:53:06.374 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:06,371 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:06.374 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:06,371 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:06.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,371 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:06.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,371 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=12850991091947631003 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:06.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,372 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:06.375 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,372 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=7338925126946267177 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:06.376 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,372 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:06.376 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,372 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:06.376 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,372 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:06.376 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:06,372 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:06.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,373 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:06.377 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:06,373 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:06.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,573 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:06.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,573 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:06.576 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,573 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:06.576 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:06,573 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:06.576 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,573 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:06.576 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:06,573 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:06.975 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,973 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:06.976 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,974 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:06.976 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,974 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:06.976 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:06,974 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:06.976 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:06,974 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:06.977 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:06,974 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:07.269 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:07,268 [shard 0] alienstore - stat 2022-03-29T17:53:07.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:07,774 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:07.776 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:07,774 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:07.777 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:07,775 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:07.777 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:07,775 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:07.777 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:07,775 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:07.777 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:07,775 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:07.976 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~59s 2022-03-29T17:53:07.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2022-03-29T17:53:08.276 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:08,275 [shard 0] alienstore - stat 2022-03-29T17:53:09.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:09,281 [shard 0] alienstore - stat 2022-03-29T17:53:09.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:09,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:09.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:09,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:09.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:09,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:09.378 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:09,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:09.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:09,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:09.379 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:09,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:10.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:10,288 [shard 0] alienstore - stat 2022-03-29T17:53:11.297 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:11,295 [shard 0] alienstore - stat 2022-03-29T17:53:11.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,507 [shard 0] osd - peering_event(id=398, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:11.510 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,507 [shard 0] osd - peering_event(id=398, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:11.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,507 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:53:11.511 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,507 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:53:11.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,507 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 214.140472412s) 2022-03-29T17:53:11.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,507 [shard 0] osd - log is not dirty 2022-03-29T17:53:11.512 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,507 [shard 0] osd - peering_event(id=398, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:11.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,507 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:11.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,508 [shard 0] osd - peering_event(id=398, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:11.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,517 [shard 0] osd - peering_event(id=399, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:11.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,517 [shard 0] osd - peering_event(id=399, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:11.520 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,517 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:53:11.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,517 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:53:11.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,517 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 214.150482178s) 2022-03-29T17:53:11.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,517 [shard 0] osd - log is not dirty 2022-03-29T17:53:11.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,517 [shard 0] osd - peering_event(id=399, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:11.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,517 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:11.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,518 [shard 0] osd - peering_event(id=399, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:11.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,547 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:11.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,547 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:11.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,547 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:53:11.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,547 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:53:11.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,547 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 214.180541992s) 2022-03-29T17:53:11.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,547 [shard 0] osd - log is not dirty 2022-03-29T17:53:11.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,547 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:11.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,547 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:11.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,548 [shard 0] osd - peering_event(id=400, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:11.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,577 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:11.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,577 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:11.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,577 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:53:11.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,577 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:53:11.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,577 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 214.210723877s) 2022-03-29T17:53:11.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,577 [shard 0] osd - log is not dirty 2022-03-29T17:53:11.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,577 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:11.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,577 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:11.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,578 [shard 0] osd - peering_event(id=401, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:11.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,587 [shard 0] osd - peering_event(id=402, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:11.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,587 [shard 0] osd - peering_event(id=402, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:11.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,587 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:53:11.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,588 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:53:11.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,588 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 214.220977783s) 2022-03-29T17:53:11.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,588 [shard 0] osd - log is not dirty 2022-03-29T17:53:11.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,588 [shard 0] osd - peering_event(id=402, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:11.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,588 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:11.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,588 [shard 0] osd - peering_event(id=402, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:11.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,617 [shard 0] osd - peering_event(id=403, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:11.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,617 [shard 0] osd - peering_event(id=403, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:11.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,617 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:53:11.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,617 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:53:11.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,618 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 214.250961304s) 2022-03-29T17:53:11.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,618 [shard 0] osd - log is not dirty 2022-03-29T17:53:11.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,618 [shard 0] osd - peering_event(id=403, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:11.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,618 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:11.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:11,618 [shard 0] osd - peering_event(id=403, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:12.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:12.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:12.256 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:12.257 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:12.257 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:12.257 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:12.258 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:53:12.258 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 214.885894775s (sent 214.885894775s) 2022-03-29T17:53:12.258 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - log is not dirty 2022-03-29T17:53:12.259 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:12.259 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,252 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:12.259 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:53:12.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 214.886123657s (sent 214.886123657s) 2022-03-29T17:53:12.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - log is not dirty 2022-03-29T17:53:12.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:12.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:12.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:53:12.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 214.886322021s (sent 214.886322021s) 2022-03-29T17:53:12.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - log is not dirty 2022-03-29T17:53:12.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:12.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:12.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - peering_event(id=405, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:12.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - peering_event(id=404, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:12.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,253 [shard 0] osd - peering_event(id=406, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:12.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,272 [shard 0] osd - peering_event(id=407, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:12.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,272 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:12.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,272 [shard 0] osd - peering_event(id=409, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:12.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,272 [shard 0] osd - peering_event(id=407, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:12.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,272 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:12.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,272 [shard 0] osd - peering_event(id=409, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:12.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,272 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:53:12.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 214.906005859s (sent 214.906005859s) 2022-03-29T17:53:12.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - log is not dirty 2022-03-29T17:53:12.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - peering_event(id=407, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:12.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:12.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:53:12.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 214.906158447s (sent 214.906158447s) 2022-03-29T17:53:12.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - log is not dirty 2022-03-29T17:53:12.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - peering_event(id=409, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:12.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:12.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:53:12.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 214.906387329s (sent 214.906387329s) 2022-03-29T17:53:12.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - log is not dirty 2022-03-29T17:53:12.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:12.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:12.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,273 [shard 0] osd - peering_event(id=409, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:12.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,274 [shard 0] osd - peering_event(id=407, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:12.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:12,274 [shard 0] osd - peering_event(id=408, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:12.304 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,302 [shard 0] alienstore - stat 2022-03-29T17:53:12.375 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:12,372 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:12.376 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:12,373 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:12.376 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,373 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:12.376 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,373 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=14915683259733792077 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:12.376 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,373 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:12.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,373 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=3662348653613317768 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:12.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,373 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:12.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,373 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:12.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:12.377 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:12,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:12.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:12.378 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:12,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:12.576 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,574 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:12.577 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,574 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:12.577 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,574 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:12.577 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:12,575 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:12.577 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,575 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:12.578 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:12,575 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:12.977 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,975 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:12.977 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,975 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:12.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,975 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:12.978 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:12,975 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:12.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:12,975 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:12.978 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:12,975 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:13.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2022-03-29T17:53:13.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~37s 2022-03-29T17:53:13.310 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:13,309 [shard 0] alienstore - stat 2022-03-29T17:53:13.777 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:13,775 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:13.777 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:13,776 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:13.778 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:13,776 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:13.779 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:13,776 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:13.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:13,776 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:13.779 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:13,776 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:13.895 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,893 [shard 0] osd - peering_event(id=410, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:13.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,893 [shard 0] osd - peering_event(id=410, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:13.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,893 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:53:13.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,893 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:53:13.896 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,893 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 216.526855469s) 2022-03-29T17:53:13.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,893 [shard 0] osd - log is not dirty 2022-03-29T17:53:13.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,893 [shard 0] osd - peering_event(id=410, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:13.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,894 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:13.897 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,894 [shard 0] osd - peering_event(id=410, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:13.904 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,903 [shard 0] osd - peering_event(id=411, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:13.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,904 [shard 0] osd - peering_event(id=411, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:13.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,904 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:53:13.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,904 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:53:13.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,904 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 216.537277222s) 2022-03-29T17:53:13.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,904 [shard 0] osd - log is not dirty 2022-03-29T17:53:13.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,904 [shard 0] osd - peering_event(id=411, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:13.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,904 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:13.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:13,905 [shard 0] osd - peering_event(id=411, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:14.280 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:14,279 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:14.281 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:14,279 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:14.281 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:14,279 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:14.281 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:14,279 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:53:14.282 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:14,279 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:14.318 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:14,316 [shard 0] alienstore - stat 2022-03-29T17:53:15.324 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:15,323 [shard 0] alienstore - stat 2022-03-29T17:53:15.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:15,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:15.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:15,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:15.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:15,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:15.380 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:15,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:15.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:15,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:15.380 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:15,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:16.331 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:16,329 [shard 0] alienstore - stat 2022-03-29T17:53:17.338 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:17,336 [shard 0] alienstore - stat 2022-03-29T17:53:17.865 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:17,863 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:53:17.866 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:17,864 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:53:17.866 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:17,864 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:17.866 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:17,864 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:53:17.867 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:17,864 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:53:18.345 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,343 [shard 0] alienstore - stat 2022-03-29T17:53:18.377 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:18,373 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:18.378 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:18,373 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:18.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,373 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:18.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,373 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=5282559486289548973 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:18.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,373 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:18.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,373 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=17625408280129538978 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:18.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:18.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:18.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:18.381 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:18,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:18.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:18.382 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:18,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:18.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,574 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:18.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,574 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:18.581 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,575 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:18.581 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:18,575 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:18.582 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,575 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:18.582 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:18,575 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:18.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2022-03-29T17:53:18.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~42s 2022-03-29T17:53:18.977 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,975 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:18.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,975 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:18.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,976 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:18.978 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:18,976 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:18.979 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:18,976 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:18.979 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:18,976 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:19.351 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:19,350 [shard 0] alienstore - stat 2022-03-29T17:53:19.513 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,510 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:19.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,510 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:19.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,511 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:53:19.514 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,511 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:53:19.515 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,511 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 222.144104004s) 2022-03-29T17:53:19.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,511 [shard 0] osd - log is not dirty 2022-03-29T17:53:19.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,511 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:19.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,511 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:19.516 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,512 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:19.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,520 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:19.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,520 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:19.524 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,521 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:53:19.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,521 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:53:19.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,521 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 222.154129028s) 2022-03-29T17:53:19.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,521 [shard 0] osd - log is not dirty 2022-03-29T17:53:19.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,521 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:19.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,521 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:19.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,521 [shard 0] osd - peering_event(id=413, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:19.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,551 [shard 0] osd - peering_event(id=414, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:19.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,551 [shard 0] osd - peering_event(id=414, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:19.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,551 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:53:19.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,551 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:53:19.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,551 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 222.184478760s) 2022-03-29T17:53:19.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,551 [shard 0] osd - log is not dirty 2022-03-29T17:53:19.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,551 [shard 0] osd - peering_event(id=414, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:19.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,551 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:19.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,552 [shard 0] osd - peering_event(id=414, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:19.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,581 [shard 0] osd - peering_event(id=415, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:19.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,581 [shard 0] osd - peering_event(id=415, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:19.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,581 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:53:19.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,581 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:53:19.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,581 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 222.214599609s) 2022-03-29T17:53:19.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,581 [shard 0] osd - log is not dirty 2022-03-29T17:53:19.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,581 [shard 0] osd - peering_event(id=415, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:19.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,581 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:19.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,582 [shard 0] osd - peering_event(id=415, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:19.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,591 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:19.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,591 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:19.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,591 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:53:19.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,591 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:53:19.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,591 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 222.224731445s) 2022-03-29T17:53:19.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,591 [shard 0] osd - log is not dirty 2022-03-29T17:53:19.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,591 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:19.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,591 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:19.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,592 [shard 0] osd - peering_event(id=416, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:19.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,621 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:19.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,621 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:19.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,621 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:53:19.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,621 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:53:19.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,621 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 222.254928589s) 2022-03-29T17:53:19.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,621 [shard 0] osd - log is not dirty 2022-03-29T17:53:19.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,621 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:19.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,622 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:19.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:19,622 [shard 0] osd - peering_event(id=417, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:19.778 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:19,776 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:19.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:19,776 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:19.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:19,777 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:19.779 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:19,777 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:19.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:19,777 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:19.781 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:19,777 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:20.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=418, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:20.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:20.260 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:20.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:20.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:20.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=418, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:20.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:53:20.261 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 222.889572144s (sent 222.889572144s) 2022-03-29T17:53:20.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - log is not dirty 2022-03-29T17:53:20.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:20.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:20.262 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:53:20.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 222.889755249s (sent 222.889755249s) 2022-03-29T17:53:20.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - log is not dirty 2022-03-29T17:53:20.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:20.263 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:20.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:53:20.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 222.889862061s (sent 222.889862061s) 2022-03-29T17:53:20.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - log is not dirty 2022-03-29T17:53:20.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - peering_event(id=418, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:20.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,256 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:20.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,257 [shard 0] osd - peering_event(id=420, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:20.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,257 [shard 0] osd - peering_event(id=418, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:20.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,257 [shard 0] osd - peering_event(id=419, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:20.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:20.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:20.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:20.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:20.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:20.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:20.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:53:20.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 222.909637451s (sent 222.909637451s) 2022-03-29T17:53:20.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - log is not dirty 2022-03-29T17:53:20.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:20.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:20.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:53:20.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 222.909835815s (sent 222.909835815s) 2022-03-29T17:53:20.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - log is not dirty 2022-03-29T17:53:20.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:20.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:20.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,276 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:53:20.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,277 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 222.910018921s (sent 222.910018921s) 2022-03-29T17:53:20.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,277 [shard 0] osd - log is not dirty 2022-03-29T17:53:20.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,277 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:20.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,277 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:20.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,277 [shard 0] osd - peering_event(id=423, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:20.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,277 [shard 0] osd - peering_event(id=422, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:20.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:20,277 [shard 0] osd - peering_event(id=421, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:20.358 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:20,356 [shard 0] alienstore - stat 2022-03-29T17:53:21.365 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:21,363 [shard 0] alienstore - stat 2022-03-29T17:53:21.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:21,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:21.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:21,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:21.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:21,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:21.380 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:21,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:21.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:21,378 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:21.380 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:21,378 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:21.899 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,897 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:21.900 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,897 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:21.900 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,897 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:53:21.900 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,897 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:53:21.900 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,897 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 224.530654907s) 2022-03-29T17:53:21.900 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,897 [shard 0] osd - log is not dirty 2022-03-29T17:53:21.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,897 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:21.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,897 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:21.901 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,898 [shard 0] osd - peering_event(id=424, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:21.909 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,907 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:21.909 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,907 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:21.910 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,907 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:53:21.910 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,907 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:53:21.910 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,907 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 224.540710449s) 2022-03-29T17:53:21.910 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,907 [shard 0] osd - log is not dirty 2022-03-29T17:53:21.911 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,907 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:21.911 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,907 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:21.911 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:21,908 [shard 0] osd - peering_event(id=425, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:22.372 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:22,370 [shard 0] alienstore - stat 2022-03-29T17:53:23.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:23,377 [shard 0] alienstore - stat 2022-03-29T17:53:23.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~75s 2022-03-29T17:53:23.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~48s 2022-03-29T17:53:24.376 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:24,374 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:24.377 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:24,374 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:24.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:24.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=2535377977740318949 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:24.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:24.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=9588393039825872503 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:24.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:24.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:24.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:24.379 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:24,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:24.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:24.380 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:24,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:24.385 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,384 [shard 0] alienstore - stat 2022-03-29T17:53:24.577 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,575 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:24.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,575 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:24.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,576 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:24.578 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:24,576 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:24.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,576 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:24.579 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:24,576 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:24.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,976 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:24.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,976 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:24.979 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,976 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:24.979 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:24,976 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:24.979 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:24,976 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:24.979 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:24,977 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:25.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:25,390 [shard 0] alienstore - stat 2022-03-29T17:53:25.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:25,777 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:25.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:25,777 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:25.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:25,777 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:25.780 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:25,777 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:25.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:25,777 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:25.780 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:25,777 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:26.399 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:26,397 [shard 0] alienstore - stat 2022-03-29T17:53:27.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:27,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:27.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:27,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:27.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:27,378 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:27.380 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:27,378 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:27.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:27,378 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:27.381 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:27,378 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:27.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:27,404 [shard 0] alienstore - stat 2022-03-29T17:53:27.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,515 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:27.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,515 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:27.517 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,515 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:53:27.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,515 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:53:27.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,515 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 230.148376465s) 2022-03-29T17:53:27.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,515 [shard 0] osd - log is not dirty 2022-03-29T17:53:27.518 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,515 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:27.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,515 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:27.519 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,516 [shard 0] osd - peering_event(id=426, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:27.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,525 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:27.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,525 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:27.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,525 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:53:27.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,525 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:53:27.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,525 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 230.158493042s) 2022-03-29T17:53:27.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,525 [shard 0] osd - log is not dirty 2022-03-29T17:53:27.528 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,525 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:27.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,525 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:27.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,526 [shard 0] osd - peering_event(id=427, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:27.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,555 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:27.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,555 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:27.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,555 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:53:27.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,555 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:53:27.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,555 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 230.188705444s) 2022-03-29T17:53:27.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,555 [shard 0] osd - log is not dirty 2022-03-29T17:53:27.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,555 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:27.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,555 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:27.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,556 [shard 0] osd - peering_event(id=428, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:27.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,585 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:27.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,585 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:27.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,585 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:53:27.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,585 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:53:27.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,585 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 230.218902588s) 2022-03-29T17:53:27.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,585 [shard 0] osd - log is not dirty 2022-03-29T17:53:27.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,585 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:27.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,585 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:27.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,586 [shard 0] osd - peering_event(id=429, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:27.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,595 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:27.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,595 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:27.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,595 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:53:27.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,596 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:53:27.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,596 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 230.229019165s) 2022-03-29T17:53:27.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,596 [shard 0] osd - log is not dirty 2022-03-29T17:53:27.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,596 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:27.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,596 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:27.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,596 [shard 0] osd - peering_event(id=430, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:27.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,625 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:27.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,625 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:27.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,626 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:53:27.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,626 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:53:27.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,626 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 230.259094238s) 2022-03-29T17:53:27.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,626 [shard 0] osd - log is not dirty 2022-03-29T17:53:27.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,626 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:27.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,626 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:27.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:27,626 [shard 0] osd - peering_event(id=431, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:28.264 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:28.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:28.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:28.265 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:28.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:28.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:28.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:53:28.266 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 230.893676758s (sent 230.893676758s) 2022-03-29T17:53:28.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - log is not dirty 2022-03-29T17:53:28.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:28.267 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:28.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:53:28.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 230.893951416s (sent 230.893951416s) 2022-03-29T17:53:28.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - log is not dirty 2022-03-29T17:53:28.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:28.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,260 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:28.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,261 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:53:28.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,261 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 230.894088745s (sent 230.894088745s) 2022-03-29T17:53:28.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,261 [shard 0] osd - log is not dirty 2022-03-29T17:53:28.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,261 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:28.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,261 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:28.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,261 [shard 0] osd - peering_event(id=433, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:28.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,261 [shard 0] osd - peering_event(id=434, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:28.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,261 [shard 0] osd - peering_event(id=432, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:28.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:28.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:28.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:28.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:28.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:28.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:28.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:53:28.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 230.913635254s (sent 230.913635254s) 2022-03-29T17:53:28.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - log is not dirty 2022-03-29T17:53:28.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:28.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:28.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:53:28.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 230.913864136s (sent 230.913864136s) 2022-03-29T17:53:28.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - log is not dirty 2022-03-29T17:53:28.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:28.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,280 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:28.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,281 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:53:28.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,281 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 230.914062500s (sent 230.914062500s) 2022-03-29T17:53:28.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,281 [shard 0] osd - log is not dirty 2022-03-29T17:53:28.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,281 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:28.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,281 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:28.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,281 [shard 0] osd - peering_event(id=436, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:28.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,281 [shard 0] osd - peering_event(id=435, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:28.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:28,281 [shard 0] osd - peering_event(id=437, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:28.412 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:28,411 [shard 0] alienstore - stat 2022-03-29T17:53:29.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2022-03-29T17:53:29.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~53s 2022-03-29T17:53:29.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:29,279 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:29.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:29,279 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:29.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:29,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:29.284 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:29,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:53:29.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:29,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:29.419 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:29,418 [shard 0] alienstore - stat 2022-03-29T17:53:29.904 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,901 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:29.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,901 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:29.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,901 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:53:29.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,901 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:53:29.905 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,901 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 232.534591675s) 2022-03-29T17:53:29.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,901 [shard 0] osd - log is not dirty 2022-03-29T17:53:29.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,901 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:29.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,901 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:29.906 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,902 [shard 0] osd - peering_event(id=438, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:29.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,911 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:29.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,911 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:29.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,911 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:53:29.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,911 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:53:29.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,911 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 232.544494629s) 2022-03-29T17:53:29.914 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,911 [shard 0] osd - log is not dirty 2022-03-29T17:53:29.915 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,911 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:29.915 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,911 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:29.915 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:29,912 [shard 0] osd - peering_event(id=439, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:30.378 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:30,374 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:30.378 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:30,374 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:30.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:30.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,374 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=16861991748452978107 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:30.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:30.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,374 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=248566943256161178 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:30.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:30.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:30.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:30.382 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:30,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:30.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:30.383 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:30,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:30.426 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,424 [shard 0] alienstore - stat 2022-03-29T17:53:30.577 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,575 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:30.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,576 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:30.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,576 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:30.578 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:30,576 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:30.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,576 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:30.579 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:30,576 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:30.978 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,976 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:30.979 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,976 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:30.979 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,977 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:30.979 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:30,977 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:30.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:30,977 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:30.980 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:30,977 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:31.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:53:31.211+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:53:31.433 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:31,431 [shard 0] alienstore - stat 2022-03-29T17:53:31.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:31,777 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:31.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:31,777 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:31.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:31,777 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:31.780 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:31,777 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:31.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:31,778 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:31.780 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:31,778 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:32.439 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:32,438 [shard 0] alienstore - stat 2022-03-29T17:53:32.866 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:32,864 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:53:32.867 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:32,864 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:53:32.867 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:32,864 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:32.867 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:32,865 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:53:32.867 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:32,865 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:53:33.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:33,378 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:33.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:33,378 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:33.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:33,378 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:33.381 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:33,378 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:33.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:33,378 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:33.382 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:33,378 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:33.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:33,445 [shard 0] alienstore - stat 2022-03-29T17:53:34.454 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:34,452 [shard 0] alienstore - stat 2022-03-29T17:53:34.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2022-03-29T17:53:34.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~58s 2022-03-29T17:53:35.460 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:35,459 [shard 0] alienstore - stat 2022-03-29T17:53:35.521 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,519 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:35.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,519 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:35.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,519 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:53:35.522 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,519 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:53:35.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,519 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 238.152709961s) 2022-03-29T17:53:35.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,519 [shard 0] osd - log is not dirty 2022-03-29T17:53:35.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,519 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:35.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,519 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:35.523 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,520 [shard 0] osd - peering_event(id=440, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:35.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,529 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:35.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,529 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:35.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,529 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:53:35.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,529 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:53:35.532 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,529 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 238.162628174s) 2022-03-29T17:53:35.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,529 [shard 0] osd - log is not dirty 2022-03-29T17:53:35.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,529 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:35.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,529 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:35.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,530 [shard 0] osd - peering_event(id=441, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:35.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,559 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:35.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,559 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:35.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,559 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:53:35.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,559 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:53:35.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,560 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 238.192932129s) 2022-03-29T17:53:35.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,560 [shard 0] osd - log is not dirty 2022-03-29T17:53:35.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,560 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:35.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,560 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:35.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,560 [shard 0] osd - peering_event(id=442, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:35.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,589 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:35.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,589 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:35.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,589 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:53:35.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,590 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:53:35.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,590 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 238.223037720s) 2022-03-29T17:53:35.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,590 [shard 0] osd - log is not dirty 2022-03-29T17:53:35.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,590 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:35.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,590 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:35.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,590 [shard 0] osd - peering_event(id=443, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:35.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,599 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:35.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,599 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:35.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,600 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:53:35.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,600 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:53:35.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,600 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 238.233078003s) 2022-03-29T17:53:35.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,600 [shard 0] osd - log is not dirty 2022-03-29T17:53:35.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,600 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:35.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,600 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:35.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,600 [shard 0] osd - peering_event(id=444, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:35.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,629 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:35.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,630 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:35.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,630 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:53:35.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,630 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:53:35.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,630 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 238.263381958s) 2022-03-29T17:53:35.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,630 [shard 0] osd - log is not dirty 2022-03-29T17:53:35.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,630 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:35.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,630 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:35.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:35,631 [shard 0] osd - peering_event(id=445, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:36.215 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:53:36.212+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:53:36.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,264 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:36.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,264 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:36.268 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,264 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:36.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,264 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:36.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,264 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:36.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,264 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:36.269 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:53:36.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 238.898147583s (sent 238.898147583s) 2022-03-29T17:53:36.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - log is not dirty 2022-03-29T17:53:36.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:36.270 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:36.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:53:36.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 238.898391724s (sent 238.898391724s) 2022-03-29T17:53:36.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - log is not dirty 2022-03-29T17:53:36.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:36.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:36.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:53:36.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 238.898574829s (sent 238.898574829s) 2022-03-29T17:53:36.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - log is not dirty 2022-03-29T17:53:36.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:36.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,265 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:36.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,266 [shard 0] osd - peering_event(id=448, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:36.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,266 [shard 0] osd - peering_event(id=447, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:36.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,266 [shard 0] osd - peering_event(id=446, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:36.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,284 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:36.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,284 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:36.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,284 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:36.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,284 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:36.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,284 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:36.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:36.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:53:36.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 238.918212891s (sent 238.918212891s) 2022-03-29T17:53:36.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - log is not dirty 2022-03-29T17:53:36.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:36.290 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:36.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:53:36.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 238.918380737s (sent 238.918380737s) 2022-03-29T17:53:36.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - log is not dirty 2022-03-29T17:53:36.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:36.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:36.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:53:36.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 238.918624878s (sent 238.918624878s) 2022-03-29T17:53:36.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - log is not dirty 2022-03-29T17:53:36.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:36.293 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,285 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:36.293 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,286 [shard 0] osd - peering_event(id=449, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:36.293 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,286 [shard 0] osd - peering_event(id=450, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:36.293 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:36,286 [shard 0] osd - peering_event(id=451, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:36.379 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:36,375 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:36.379 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:36,375 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:36.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:36.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=5073992187123482892 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:36.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:36.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=16399069491941957374 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:36.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:36.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:36.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:36.383 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:36,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:36.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:36.384 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:36,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:36.467 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,466 [shard 0] alienstore - stat 2022-03-29T17:53:36.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,576 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:36.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,576 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:36.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,577 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:36.579 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:36,577 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:36.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,577 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:36.580 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:36,577 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:36.979 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,977 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:36.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,977 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:36.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,977 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:36.980 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:36,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:36.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:36,978 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:36.981 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:36,978 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:37.474 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:37,472 [shard 0] alienstore - stat 2022-03-29T17:53:37.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:37,778 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:37.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:37,778 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:37.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:37,778 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:37.781 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:37,778 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:37.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:37,778 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:37.781 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:37,778 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:37.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,905 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:37.907 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,905 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:37.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,905 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:53:37.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,905 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:53:37.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,905 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 240.538452148s) 2022-03-29T17:53:37.908 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,905 [shard 0] osd - log is not dirty 2022-03-29T17:53:37.909 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,905 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:37.909 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,905 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:37.910 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,906 [shard 0] osd - peering_event(id=452, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:37.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,915 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:37.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,915 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:37.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,915 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:53:37.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,915 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:53:37.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,915 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 240.548583984s) 2022-03-29T17:53:37.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,915 [shard 0] osd - log is not dirty 2022-03-29T17:53:37.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,915 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:37.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,915 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:37.919 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:37,916 [shard 0] osd - peering_event(id=453, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:38.481 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:38,479 [shard 0] alienstore - stat 2022-03-29T17:53:39.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:39,378 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:39.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:39,379 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:39.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:39,379 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:39.381 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:39,379 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:39.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:39,379 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:39.382 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:39,379 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:39.487 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:39,486 [shard 0] alienstore - stat 2022-03-29T17:53:39.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~91s 2022-03-29T17:53:39.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~64s 2022-03-29T17:53:40.494 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:40,492 [shard 0] alienstore - stat 2022-03-29T17:53:41.216 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:53:41.213+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:53:41.501 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:41,499 [shard 0] alienstore - stat 2022-03-29T17:53:42.378 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:42,375 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:42.378 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:42,375 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:42.378 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:42.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,375 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=6551662765270635355 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:42.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:42.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,375 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=17489222999136383401 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:42.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:42.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:42.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:42.380 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:42,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:42.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:42.381 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:42,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:42.507 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,506 [shard 0] alienstore - stat 2022-03-29T17:53:42.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,576 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:42.578 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,576 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:42.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,577 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:42.579 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:42,577 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:42.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,577 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:42.580 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:42,577 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:42.979 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,977 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:42.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:42.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,978 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:42.980 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:42,978 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:42.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:42,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:42.981 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:42,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:43.514 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:43,513 [shard 0] alienstore - stat 2022-03-29T17:53:43.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,523 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:43.525 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,523 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:43.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,523 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:53:43.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,523 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:53:43.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,523 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 246.156631470s) 2022-03-29T17:53:43.526 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,523 [shard 0] osd - log is not dirty 2022-03-29T17:53:43.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,523 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:43.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,523 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:43.527 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,524 [shard 0] osd - peering_event(id=454, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:43.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,533 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:43.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,533 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:43.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,533 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:53:43.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,533 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:53:43.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,533 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 246.166595459s) 2022-03-29T17:53:43.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,533 [shard 0] osd - log is not dirty 2022-03-29T17:53:43.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,533 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:43.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,533 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:43.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,534 [shard 0] osd - peering_event(id=455, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:43.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,563 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:43.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,563 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:43.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,563 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:53:43.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,563 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:53:43.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,563 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 246.196777344s) 2022-03-29T17:53:43.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,563 [shard 0] osd - log is not dirty 2022-03-29T17:53:43.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,563 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:43.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,563 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:43.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,564 [shard 0] osd - peering_event(id=456, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:43.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,593 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:43.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,593 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:43.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,593 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:53:43.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,593 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:53:43.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,593 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 246.226913452s) 2022-03-29T17:53:43.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,593 [shard 0] osd - log is not dirty 2022-03-29T17:53:43.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,593 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:43.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,594 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:43.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,594 [shard 0] osd - peering_event(id=457, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:43.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,603 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:43.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,603 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:43.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,603 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:53:43.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,603 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:53:43.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,603 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 246.236862183s) 2022-03-29T17:53:43.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,603 [shard 0] osd - log is not dirty 2022-03-29T17:53:43.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,603 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:43.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,603 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:43.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,604 [shard 0] osd - peering_event(id=458, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:43.637 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,633 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:43.637 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,633 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:43.637 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,634 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:53:43.637 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,634 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:53:43.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,634 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 246.267318726s) 2022-03-29T17:53:43.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,634 [shard 0] osd - log is not dirty 2022-03-29T17:53:43.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,634 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:43.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,634 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:43.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:43,635 [shard 0] osd - peering_event(id=459, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:43.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:43,778 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:43.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:43,778 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:43.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:43,778 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:43.781 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:43,779 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:43.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:43,779 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:43.781 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:43,779 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:44.271 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:44.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:44.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:44.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:44.272 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:44.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:44.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:53:44.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 246.901657104s (sent 246.901657104s) 2022-03-29T17:53:44.273 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - log is not dirty 2022-03-29T17:53:44.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:44.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:44.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:53:44.274 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 246.901947021s (sent 246.901947021s) 2022-03-29T17:53:44.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - log is not dirty 2022-03-29T17:53:44.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:44.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,268 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:44.275 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,269 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:53:44.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,269 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 246.902175903s (sent 246.902175903s) 2022-03-29T17:53:44.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,269 [shard 0] osd - log is not dirty 2022-03-29T17:53:44.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,269 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:44.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,269 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:44.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,269 [shard 0] osd - peering_event(id=460, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:44.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,270 [shard 0] osd - peering_event(id=461, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:44.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,270 [shard 0] osd - peering_event(id=462, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:44.282 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:44,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:44.282 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:44,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:44.282 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:44,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:44.282 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:44,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:53:44.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:44,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:44.291 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:44.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:44.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:44.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:44.292 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:44.293 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:44.293 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:53:44.293 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 246.921798706s (sent 246.921798706s) 2022-03-29T17:53:44.293 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - log is not dirty 2022-03-29T17:53:44.294 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:44.294 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:44.294 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:53:44.294 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 246.922012329s (sent 246.922012329s) 2022-03-29T17:53:44.294 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - log is not dirty 2022-03-29T17:53:44.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:44.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:44.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:53:44.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 246.922180176s (sent 246.922180176s) 2022-03-29T17:53:44.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - log is not dirty 2022-03-29T17:53:44.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:44.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:44.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,289 [shard 0] osd - peering_event(id=463, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:44.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,290 [shard 0] osd - peering_event(id=464, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:44.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:44,290 [shard 0] osd - peering_event(id=465, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:44.521 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:44,520 [shard 0] alienstore - stat 2022-03-29T17:53:45.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~96s 2022-03-29T17:53:45.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~69s 2022-03-29T17:53:45.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:45,379 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:45.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:45,379 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:45.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:45,379 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:45.382 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:45,380 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:45.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:45,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:45.383 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:45,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:45.528 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:45,526 [shard 0] alienstore - stat 2022-03-29T17:53:45.911 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,909 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:45.911 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,909 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:45.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,909 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:53:45.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,909 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:53:45.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,909 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 248.542343140s) 2022-03-29T17:53:45.912 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,909 [shard 0] osd - log is not dirty 2022-03-29T17:53:45.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,909 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:45.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,909 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:45.913 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,910 [shard 0] osd - peering_event(id=466, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:45.921 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,919 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:45.921 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,919 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:45.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,919 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:53:45.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,919 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:53:45.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,919 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 248.552551270s) 2022-03-29T17:53:45.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,919 [shard 0] osd - log is not dirty 2022-03-29T17:53:45.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,919 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:45.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,919 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:45.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:45,920 [shard 0] osd - peering_event(id=467, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:46.216 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:53:46.215+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:53:46.535 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:46,533 [shard 0] alienstore - stat 2022-03-29T17:53:47.541 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:47,540 [shard 0] alienstore - stat 2022-03-29T17:53:47.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:47,865 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:53:47.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:47,865 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:53:47.869 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:47,865 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:47.869 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:47,865 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:53:47.870 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:47,865 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:53:48.379 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:48,375 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:48.380 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:48,376 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:48.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:48.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=16111010893680391410 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:48.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:48.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=5455926403295692755 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:48.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:48.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:48.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:48.384 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:48,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:48.384 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:48.384 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:48,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:48.548 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,547 [shard 0] alienstore - stat 2022-03-29T17:53:48.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,577 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:48.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,577 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:48.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,577 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:48.580 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:48,577 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:48.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,577 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:48.580 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:48,578 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:48.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,977 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:48.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:48.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:48.980 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:48,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:48.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:48,978 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:48.981 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:48,978 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:49.555 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:49,554 [shard 0] alienstore - stat 2022-03-29T17:53:49.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:49,778 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:49.780 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:49,778 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:49.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:49,779 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:49.782 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:49,779 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:49.782 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:49,779 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:49.782 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:49,779 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:50.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2022-03-29T17:53:50.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~74s 2022-03-29T17:53:50.562 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:50,560 [shard 0] alienstore - stat 2022-03-29T17:53:51.218 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:53:51.215+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:53:51.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:51,379 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:51.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:51,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:51.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:51,380 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:51.383 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:51,380 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:51.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:51,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:51.383 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:51,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:51.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,527 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:51.529 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,527 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:51.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,527 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:53:51.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,527 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:53:51.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,527 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 254.160507202s) 2022-03-29T17:53:51.530 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,527 [shard 0] osd - log is not dirty 2022-03-29T17:53:51.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,527 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:51.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,527 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:51.531 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,528 [shard 0] osd - peering_event(id=468, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:51.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,537 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:51.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,537 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:51.539 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,537 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:53:51.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,537 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:53:51.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,537 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 254.170532227s) 2022-03-29T17:53:51.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,537 [shard 0] osd - log is not dirty 2022-03-29T17:53:51.540 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,537 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:51.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,537 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:51.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,538 [shard 0] osd - peering_event(id=469, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:51.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,567 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:51.570 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:51,567 [shard 0] alienstore - stat 2022-03-29T17:53:51.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,567 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:51.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,568 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:53:51.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,568 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:53:51.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,568 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 254.201080322s) 2022-03-29T17:53:51.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,568 [shard 0] osd - log is not dirty 2022-03-29T17:53:51.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,568 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:51.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,568 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:51.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,568 [shard 0] osd - peering_event(id=470, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:51.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,597 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:51.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,597 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:51.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,597 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:53:51.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,597 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:53:51.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,598 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 254.230941772s) 2022-03-29T17:53:51.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,598 [shard 0] osd - log is not dirty 2022-03-29T17:53:51.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,598 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:51.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,598 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:51.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,598 [shard 0] osd - peering_event(id=471, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:51.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,607 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:51.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,607 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:51.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,607 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:53:51.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,607 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:53:51.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,607 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 254.240844727s) 2022-03-29T17:53:51.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,607 [shard 0] osd - log is not dirty 2022-03-29T17:53:51.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,607 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:51.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,607 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:51.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,608 [shard 0] osd - peering_event(id=472, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:51.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,637 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:51.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,637 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:51.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,637 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:53:51.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,637 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:53:51.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,638 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 254.270996094s) 2022-03-29T17:53:51.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,638 [shard 0] osd - log is not dirty 2022-03-29T17:53:51.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,638 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:51.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,638 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:51.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:51,638 [shard 0] osd - peering_event(id=473, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:52.276 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - peering_event(id=474, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:52.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - peering_event(id=475, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:52.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - peering_event(id=476, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:52.277 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - peering_event(id=474, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:52.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - peering_event(id=476, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:52.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - peering_event(id=475, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:52.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:53:52.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 254.905685425s (sent 254.905685425s) 2022-03-29T17:53:52.278 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - log is not dirty 2022-03-29T17:53:52.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - peering_event(id=475, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:52.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:52.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:53:52.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 254.905944824s (sent 254.905944824s) 2022-03-29T17:53:52.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - log is not dirty 2022-03-29T17:53:52.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,272 [shard 0] osd - peering_event(id=474, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:52.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:52.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,273 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:53:52.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,273 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 254.906188965s (sent 254.906188965s) 2022-03-29T17:53:52.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,273 [shard 0] osd - log is not dirty 2022-03-29T17:53:52.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,273 [shard 0] osd - peering_event(id=476, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:52.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,273 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:52.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,273 [shard 0] osd - peering_event(id=475, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:52.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,273 [shard 0] osd - peering_event(id=474, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:52.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,274 [shard 0] osd - peering_event(id=476, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:52.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - peering_event(id=477, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:52.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - peering_event(id=478, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:52.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:53:52.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:52.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - peering_event(id=477, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:52.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - peering_event(id=478, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:53:52.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:53:52.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 254.925765991s (sent 254.925765991s) 2022-03-29T17:53:52.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - log is not dirty 2022-03-29T17:53:52.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:52.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:52.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:53:52.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 254.925964355s (sent 254.925964355s) 2022-03-29T17:53:52.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - log is not dirty 2022-03-29T17:53:52.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,292 [shard 0] osd - peering_event(id=477, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:52.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:52.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:53:52.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 254.926162720s (sent 254.926162720s) 2022-03-29T17:53:52.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - log is not dirty 2022-03-29T17:53:52.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - peering_event(id=478, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:53:52.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:52.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - peering_event(id=479, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:52.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - peering_event(id=477, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:52.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:52,293 [shard 0] osd - peering_event(id=478, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:53:52.575 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:52,574 [shard 0] alienstore - stat 2022-03-29T17:53:53.582 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:53,581 [shard 0] alienstore - stat 2022-03-29T17:53:53.916 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,913 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:53.916 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,913 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:53.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,913 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:53:53.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,913 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:53:53.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,913 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 256.546600342s) 2022-03-29T17:53:53.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,913 [shard 0] osd - log is not dirty 2022-03-29T17:53:53.917 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,913 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:53.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,913 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:53.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,914 [shard 0] osd - peering_event(id=480, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:53.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,923 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:53:53.925 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,923 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:53:53.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,923 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:53:53.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,923 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:53:53.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,923 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 256.556579590s) 2022-03-29T17:53:53.926 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,923 [shard 0] osd - log is not dirty 2022-03-29T17:53:53.927 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,923 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:53:53.927 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,923 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:53.927 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:53,924 [shard 0] osd - peering_event(id=481, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:53:54.378 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:53:54,376 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:53:54.379 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:54,376 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:53:54.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:53:54.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=15916314894798734916 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:54.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:53:54.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=9150081692562827571 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:53:54.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:54.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:53:54.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:54.381 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:54,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:53:54.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:54.381 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:54,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:53:54.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,577 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:54.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,577 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:54.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,578 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:54.580 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:54,578 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:53:54.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,578 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:54.581 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:54,578 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:53:54.589 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,587 [shard 0] alienstore - stat 2022-03-29T17:53:54.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:54.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,978 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:54.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,978 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:54.981 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:54,979 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:53:54.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:54,979 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:54.981 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:54,979 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:53:55.596 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:55,594 [shard 0] alienstore - stat 2022-03-29T17:53:55.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~107s 2022-03-29T17:53:55.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~80s 2022-03-29T17:53:55.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:55,779 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:55.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:55,779 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:55.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:55,779 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:55.782 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:55,779 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:53:55.782 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:55,779 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:55.782 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:55,779 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:53:56.218 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:53:56.216+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:53:56.602 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:56,601 [shard 0] alienstore - stat 2022-03-29T17:53:56.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,751 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4127.0:8 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): start 2022-03-29T17:53:56.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,751 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4127.0:8 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): in repeat 2022-03-29T17:53:56.754 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,751 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4127.0:8 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]) same_interval_since: 11 2022-03-29T17:53:56.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,751 [shard 0] osd - OpSequencer::start_op: op=9, last_started=7, last_unblocked=7, last_completed=7 2022-03-29T17:53:56.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,751 [shard 0] osd - do_recover_missing check for recovery, 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:53:56.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4127.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): got obc lock 2022-03-29T17:53:56.755 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - with_head_obc 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:53:56.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - with_head_obc: found 2:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:53:56.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:53:56.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - handling op omap-get-vals on object 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:53:56.756 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:53:56.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8 - object 2:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:53:56.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - with_head_obc: released 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:53:56.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,752 [shard 0] osd - OpSequencer::finish_op_in_order: op=9, last_started=9, last_unblocked=9, last_completed=7 2022-03-29T17:53:56.757 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - client_request(id=9, detail=m=[osd_op(client.4127.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): destroying 2022-03-29T17:53:56.759 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4127.0:9 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): start 2022-03-29T17:53:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4127.0:9 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): in repeat 2022-03-29T17:53:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4127.0:9 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]) same_interval_since: 14 2022-03-29T17:53:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - OpSequencer::start_op: op=10, last_started=8, last_unblocked=8, last_completed=8 2022-03-29T17:53:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - do_recover_missing check for recovery, 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:53:56.760 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4127.0:9 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): got obc lock 2022-03-29T17:53:56.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - with_head_obc 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:53:56.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - with_head_obc: found 3:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:53:56.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:53:56.761 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - handling op omap-get-vals on object 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:53:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,753 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:53:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,754 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:9 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8 - object 3:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:53:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,754 [shard 0] osd - with_head_obc: released 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:53:56.762 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,754 [shard 0] osd - OpSequencer::finish_op_in_order: op=10, last_started=10, last_unblocked=10, last_completed=8 2022-03-29T17:53:56.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:56,754 [shard 0] osd - client_request(id=10, detail=m=[osd_op(client.4127.0:9 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): destroying 2022-03-29T17:53:57.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:57,379 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:57.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:57,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:57.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:57,380 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:57.383 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:57,380 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:53:57.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:57,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:57.383 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:57,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:53:57.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:57,607 [shard 0] alienstore - stat 2022-03-29T17:53:58.616 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:58,614 [shard 0] alienstore - stat 2022-03-29T17:53:59.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:59,280 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:53:59.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:59,281 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:59.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:59,281 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:53:59.284 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:53:59,281 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:53:59.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:59,281 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:53:59.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,531 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:59.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,531 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:59.533 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,531 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:53:59.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,531 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:53:59.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,531 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 262.164428711s) 2022-03-29T17:53:59.534 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,531 [shard 0] osd - log is not dirty 2022-03-29T17:53:59.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,531 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:59.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,531 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:59.535 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,532 [shard 0] osd - peering_event(id=482, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:59.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,541 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:59.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,541 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:59.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,541 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:53:59.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,541 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:53:59.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,541 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 262.174468994s) 2022-03-29T17:53:59.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,541 [shard 0] osd - log is not dirty 2022-03-29T17:53:59.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,541 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:59.544 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,541 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:59.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,542 [shard 0] osd - peering_event(id=483, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:59.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,571 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:59.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,571 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:59.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,571 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:53:59.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,571 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:53:59.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,571 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 262.204681396s) 2022-03-29T17:53:59.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,571 [shard 0] osd - log is not dirty 2022-03-29T17:53:59.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,571 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:59.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,571 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:59.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,572 [shard 0] osd - peering_event(id=484, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:59.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,601 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:59.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,601 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:59.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,601 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:53:59.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,601 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:53:59.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,601 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 262.234680176s) 2022-03-29T17:53:59.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,601 [shard 0] osd - log is not dirty 2022-03-29T17:53:59.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,601 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:59.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,601 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:59.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,602 [shard 0] osd - peering_event(id=485, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:59.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,611 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:59.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,611 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:59.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,611 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:53:59.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,611 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:53:59.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,611 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 262.244903564s) 2022-03-29T17:53:59.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,611 [shard 0] osd - log is not dirty 2022-03-29T17:53:59.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,611 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:59.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,612 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:59.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,612 [shard 0] osd - peering_event(id=486, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:53:59.623 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:53:59,621 [shard 0] alienstore - stat 2022-03-29T17:53:59.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,641 [shard 0] osd - peering_event(id=487, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:53:59.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,641 [shard 0] osd - peering_event(id=487, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:53:59.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,642 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:53:59.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,642 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:53:59.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,642 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 262.275085449s) 2022-03-29T17:53:59.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,642 [shard 0] osd - log is not dirty 2022-03-29T17:53:59.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,642 [shard 0] osd - peering_event(id=487, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:53:59.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,642 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:53:59.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:53:59,642 [shard 0] osd - peering_event(id=487, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:00.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,275 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:00.279 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:00.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:00.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:00.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:00.280 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:00.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:54:00.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 262.909515381s (sent 262.909515381s) 2022-03-29T17:54:00.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - log is not dirty 2022-03-29T17:54:00.281 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:00.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:00.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:54:00.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 262.909759521s (sent 262.909759521s) 2022-03-29T17:54:00.282 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - log is not dirty 2022-03-29T17:54:00.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:00.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:00.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:54:00.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 262.909942627s (sent 262.909942627s) 2022-03-29T17:54:00.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - log is not dirty 2022-03-29T17:54:00.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:00.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,276 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:00.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,277 [shard 0] osd - peering_event(id=489, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:00.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,277 [shard 0] osd - peering_event(id=488, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:00.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,277 [shard 0] osd - peering_event(id=490, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:00.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,295 [shard 0] osd - peering_event(id=491, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:00.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:00.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:00.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:00.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:00.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - peering_event(id=491, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:00.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:54:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 262.929382324s (sent 262.929382324s) 2022-03-29T17:54:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - log is not dirty 2022-03-29T17:54:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:00.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:54:00.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 262.929626465s (sent 262.929626465s) 2022-03-29T17:54:00.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - log is not dirty 2022-03-29T17:54:00.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - peering_event(id=491, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:00.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:00.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:54:00.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 262.929840088s (sent 262.929840088s) 2022-03-29T17:54:00.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - log is not dirty 2022-03-29T17:54:00.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:00.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,296 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:00.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,297 [shard 0] osd - peering_event(id=492, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:00.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,297 [shard 0] osd - peering_event(id=491, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:00.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:00,297 [shard 0] osd - peering_event(id=493, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:00.378 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:00,376 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:00.378 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:00,376 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:00.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:00.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,376 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=14689288912908644253 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:00.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:00.379 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,376 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=7171936361448435788 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:00.380 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:00.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:00.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:00.381 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:00,377 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:00.381 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:00.382 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:00,377 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:00.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,577 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:00.579 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,578 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:00.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,578 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:00.581 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:00,578 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:00.581 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,578 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:00.581 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:00,578 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:00.629 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,628 [shard 0] alienstore - stat 2022-03-29T17:54:00.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,978 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:00.980 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,979 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:00.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,979 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:00.981 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:00,979 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:00.982 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:00,979 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:00.982 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:00,979 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:01.012 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~112s 2022-03-29T17:54:01.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~85s 2022-03-29T17:54:01.219 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:01.217+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:01.636 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:01,634 [shard 0] alienstore - stat 2022-03-29T17:54:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:01,779 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:01.781 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:01,779 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:01.782 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:01,780 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:01.782 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:01,780 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:01.783 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:01,780 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:01.783 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:01,780 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:01.918 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,916 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:01.919 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,916 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:01.919 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,916 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:54:01.919 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,916 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:54:01.920 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,916 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 264.549621582s) 2022-03-29T17:54:01.920 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,916 [shard 0] osd - log is not dirty 2022-03-29T17:54:01.920 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,916 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:01.920 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,916 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:01.921 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,917 [shard 0] osd - peering_event(id=494, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:01.928 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,926 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:01.928 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,926 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:01.928 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,926 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:54:01.929 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,926 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:54:01.929 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,926 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 264.559753418s) 2022-03-29T17:54:01.929 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,926 [shard 0] osd - log is not dirty 2022-03-29T17:54:01.929 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,926 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:01.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,926 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:01.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:01,927 [shard 0] osd - peering_event(id=495, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:02.643 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:02,641 [shard 0] alienstore - stat 2022-03-29T17:54:02.867 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:02,865 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:54:02.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:02,866 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:54:02.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:02,866 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:02.868 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:02,866 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:54:02.869 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:02,866 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:54:03.382 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:03,380 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:03.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:03,380 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:03.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:03,381 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:03.383 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:03,381 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:03.383 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:03,381 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:03.384 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:03,381 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:03.649 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:03,647 [shard 0] alienstore - stat 2022-03-29T17:54:04.656 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:04,654 [shard 0] alienstore - stat 2022-03-29T17:54:05.663 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:05,661 [shard 0] alienstore - stat 2022-03-29T17:54:06.220 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:06.217+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:06.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2022-03-29T17:54:06.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2022-03-29T17:54:06.389 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:06,386 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:06.390 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:06,386 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:06.390 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,386 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:06.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,386 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=10316119013774078401 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:06.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,386 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:06.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,386 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=6329386390151896566 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:06.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,386 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:06.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,387 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:06.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,387 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:06.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:06,387 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:06.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,387 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:06.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:06,387 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:06.589 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,587 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:06.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,587 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:06.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,588 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:06.590 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:06,588 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:06.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,588 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:06.591 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:06,588 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:06.669 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,668 [shard 0] alienstore - stat 2022-03-29T17:54:06.990 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,988 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:06.990 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,988 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:06.991 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,988 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:06.991 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:06,988 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:06.991 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:06,988 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:06.991 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:06,988 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:07.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,534 [shard 0] osd - peering_event(id=496, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:07.536 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,534 [shard 0] osd - peering_event(id=496, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:07.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,534 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:54:07.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,534 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:54:07.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,534 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 270.167694092s) 2022-03-29T17:54:07.537 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,534 [shard 0] osd - log is not dirty 2022-03-29T17:54:07.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,534 [shard 0] osd - peering_event(id=496, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:07.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,534 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:07.538 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,535 [shard 0] osd - peering_event(id=496, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:07.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,544 [shard 0] osd - peering_event(id=497, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:07.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,544 [shard 0] osd - peering_event(id=497, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:07.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,544 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:54:07.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,544 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:54:07.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,544 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 270.177520752s) 2022-03-29T17:54:07.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,544 [shard 0] osd - log is not dirty 2022-03-29T17:54:07.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,544 [shard 0] osd - peering_event(id=497, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:07.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,544 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:07.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,545 [shard 0] osd - peering_event(id=497, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:07.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,574 [shard 0] osd - peering_event(id=498, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:07.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,574 [shard 0] osd - peering_event(id=498, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:07.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,574 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:54:07.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,574 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:54:07.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,574 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 270.207855225s) 2022-03-29T17:54:07.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,574 [shard 0] osd - log is not dirty 2022-03-29T17:54:07.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,574 [shard 0] osd - peering_event(id=498, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:07.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,575 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:07.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,575 [shard 0] osd - peering_event(id=498, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:07.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,604 [shard 0] osd - peering_event(id=499, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:07.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,604 [shard 0] osd - peering_event(id=499, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:07.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,604 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:54:07.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,604 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:54:07.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,604 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 270.237884521s) 2022-03-29T17:54:07.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,604 [shard 0] osd - log is not dirty 2022-03-29T17:54:07.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,604 [shard 0] osd - peering_event(id=499, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:07.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,604 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:07.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,605 [shard 0] osd - peering_event(id=499, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:07.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,614 [shard 0] osd - peering_event(id=500, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:07.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,614 [shard 0] osd - peering_event(id=500, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:07.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,614 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:54:07.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,614 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:54:07.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,614 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 270.247863770s) 2022-03-29T17:54:07.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,614 [shard 0] osd - log is not dirty 2022-03-29T17:54:07.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,614 [shard 0] osd - peering_event(id=500, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:07.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,614 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:07.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,615 [shard 0] osd - peering_event(id=500, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,644 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,644 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,644 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:54:07.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,645 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:54:07.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,645 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 270.278015137s) 2022-03-29T17:54:07.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,645 [shard 0] osd - log is not dirty 2022-03-29T17:54:07.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,645 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:07.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,645 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:07.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:07,645 [shard 0] osd - peering_event(id=501, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:07.676 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:07,675 [shard 0] alienstore - stat 2022-03-29T17:54:07.790 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:07,789 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:07.791 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:07,789 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:07.791 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:07,789 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:07.792 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:07,789 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:07.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:07,789 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:07.793 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:07,789 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:08.283 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,278 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:08.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,278 [shard 0] osd - peering_event(id=503, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:08.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,278 [shard 0] osd - peering_event(id=504, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:08.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,278 [shard 0] osd - peering_event(id=504, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:08.284 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - peering_event(id=503, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:08.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:08.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:54:08.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 270.912261963s (sent 270.912261963s) 2022-03-29T17:54:08.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - log is not dirty 2022-03-29T17:54:08.285 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - peering_event(id=504, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:08.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:08.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:54:08.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 270.912567139s (sent 270.912567139s) 2022-03-29T17:54:08.286 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - log is not dirty 2022-03-29T17:54:08.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - peering_event(id=503, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:08.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:08.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:54:08.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 270.912750244s (sent 270.912750244s) 2022-03-29T17:54:08.287 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - log is not dirty 2022-03-29T17:54:08.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:08.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,279 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:08.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,280 [shard 0] osd - peering_event(id=504, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:08.288 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,280 [shard 0] osd - peering_event(id=502, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:08.289 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,280 [shard 0] osd - peering_event(id=503, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:08.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,298 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:08.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,298 [shard 0] osd - peering_event(id=506, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:08.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,298 [shard 0] osd - peering_event(id=507, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:08.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,298 [shard 0] osd - peering_event(id=507, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:08.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,298 [shard 0] osd - peering_event(id=506, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:08.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,298 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:08.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:54:08.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 270.932189941s (sent 270.932189941s) 2022-03-29T17:54:08.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - log is not dirty 2022-03-29T17:54:08.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - peering_event(id=506, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:08.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:08.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:54:08.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 270.932434082s (sent 270.932434082s) 2022-03-29T17:54:08.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - log is not dirty 2022-03-29T17:54:08.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - peering_event(id=507, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:08.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:08.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:54:08.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 270.932617188s (sent 270.932617188s) 2022-03-29T17:54:08.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - log is not dirty 2022-03-29T17:54:08.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:08.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:08.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,299 [shard 0] osd - peering_event(id=506, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:08.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,300 [shard 0] osd - peering_event(id=507, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:08.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:08,300 [shard 0] osd - peering_event(id=505, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:08.683 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:08,681 [shard 0] alienstore - stat 2022-03-29T17:54:09.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:09,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:09.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:09,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:09.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:09,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:09.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:09,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:09.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:09,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:09.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:09,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:09.690 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:09,688 [shard 0] alienstore - stat 2022-03-29T17:54:09.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,919 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:09.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,920 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:09.922 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,920 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:54:09.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,920 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:54:09.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,920 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 272.553405762s) 2022-03-29T17:54:09.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,920 [shard 0] osd - log is not dirty 2022-03-29T17:54:09.923 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,920 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:09.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,920 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:09.924 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,921 [shard 0] osd - peering_event(id=508, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:09.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,930 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:09.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,930 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:09.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,930 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:54:09.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,930 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:54:09.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,930 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 272.563293457s) 2022-03-29T17:54:09.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,930 [shard 0] osd - log is not dirty 2022-03-29T17:54:09.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,930 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:09.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,930 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:09.934 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:09,931 [shard 0] osd - peering_event(id=509, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:10.697 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:10,695 [shard 0] alienstore - stat 2022-03-29T17:54:11.221 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:11.218+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:11.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2022-03-29T17:54:11.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2022-03-29T17:54:11.704 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:11,702 [shard 0] alienstore - stat 2022-03-29T17:54:12.390 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:12,387 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:12.390 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:12,387 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:12.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,387 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:12.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,387 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=17750825600858114222 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:12.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,387 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:12.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,387 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=11754922492325105226 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:12.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,387 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:12.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,388 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:12.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,388 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:12.392 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:12,388 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:12.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,388 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:12.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:12,388 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:12.590 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,588 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:12.591 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,588 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:12.591 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,589 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:12.591 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:12,589 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:12.591 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,589 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:12.592 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:12,589 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:12.711 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,709 [shard 0] alienstore - stat 2022-03-29T17:54:12.992 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,989 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:12.992 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,989 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:12.992 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,990 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:12.993 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:12,990 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:12.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:12,990 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:12.993 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:12,990 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:13.718 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:13,716 [shard 0] alienstore - stat 2022-03-29T17:54:13.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:13,790 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:13.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:13,790 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:13.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:13,790 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:13.793 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:13,790 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:13.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:13,791 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:13.793 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:13,791 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:14.283 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:14,281 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:14.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:14,281 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:54:14.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:14,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:14.284 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:14,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:54:14.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:14,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:54:14.725 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:14,723 [shard 0] alienstore - stat 2022-03-29T17:54:15.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:15,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:15.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:15,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:15.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:15,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:15.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:15,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:15.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:15,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:15.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:15,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:15.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,539 [shard 0] osd - peering_event(id=510, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:15.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,539 [shard 0] osd - peering_event(id=510, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:15.541 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,539 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:54:15.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,539 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:54:15.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,539 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 278.172393799s) 2022-03-29T17:54:15.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,539 [shard 0] osd - log is not dirty 2022-03-29T17:54:15.542 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,539 [shard 0] osd - peering_event(id=510, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:15.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,539 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:15.543 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,540 [shard 0] osd - peering_event(id=510, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:15.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,549 [shard 0] osd - peering_event(id=511, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:15.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,549 [shard 0] osd - peering_event(id=511, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:15.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,549 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:54:15.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,549 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:54:15.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,549 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 278.182373047s) 2022-03-29T17:54:15.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,549 [shard 0] osd - log is not dirty 2022-03-29T17:54:15.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,549 [shard 0] osd - peering_event(id=511, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:15.552 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,549 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:15.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,550 [shard 0] osd - peering_event(id=511, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:15.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,579 [shard 0] osd - peering_event(id=512, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:15.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,579 [shard 0] osd - peering_event(id=512, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:15.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,579 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:54:15.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,579 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:54:15.582 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,579 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 278.212585449s) 2022-03-29T17:54:15.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,579 [shard 0] osd - log is not dirty 2022-03-29T17:54:15.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,579 [shard 0] osd - peering_event(id=512, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:15.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,579 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:15.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,580 [shard 0] osd - peering_event(id=512, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:15.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,609 [shard 0] osd - peering_event(id=513, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:15.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,609 [shard 0] osd - peering_event(id=513, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:15.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,609 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:54:15.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,609 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:54:15.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,609 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 278.242828369s) 2022-03-29T17:54:15.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,609 [shard 0] osd - log is not dirty 2022-03-29T17:54:15.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,609 [shard 0] osd - peering_event(id=513, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:15.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,609 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:15.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,610 [shard 0] osd - peering_event(id=513, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:15.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,619 [shard 0] osd - peering_event(id=514, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:15.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,619 [shard 0] osd - peering_event(id=514, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:15.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,619 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:54:15.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,619 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:54:15.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,619 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 278.252960205s) 2022-03-29T17:54:15.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,620 [shard 0] osd - log is not dirty 2022-03-29T17:54:15.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,620 [shard 0] osd - peering_event(id=514, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:15.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,620 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:15.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,620 [shard 0] osd - peering_event(id=514, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:15.652 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,649 [shard 0] osd - peering_event(id=515, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:15.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,650 [shard 0] osd - peering_event(id=515, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:15.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,650 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:54:15.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,650 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:54:15.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,650 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 278.283172607s) 2022-03-29T17:54:15.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,650 [shard 0] osd - log is not dirty 2022-03-29T17:54:15.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,650 [shard 0] osd - peering_event(id=515, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:15.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,650 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:15.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:15,650 [shard 0] osd - peering_event(id=515, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:15.732 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:15,730 [shard 0] alienstore - stat 2022-03-29T17:54:16.222 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:16.219+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:16.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,283 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,283 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:16.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:16.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:54:16.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 278.917480469s (sent 278.917480469s) 2022-03-29T17:54:16.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - log is not dirty 2022-03-29T17:54:16.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:16.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:16.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:54:16.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 278.917816162s (sent 278.917816162s) 2022-03-29T17:54:16.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - log is not dirty 2022-03-29T17:54:16.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:16.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:16.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,284 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:54:16.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,285 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 278.917999268s (sent 278.917999268s) 2022-03-29T17:54:16.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,285 [shard 0] osd - log is not dirty 2022-03-29T17:54:16.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,285 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:16.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,285 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:16.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,285 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:16.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,285 [shard 0] osd - peering_event(id=516, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:16.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,285 [shard 0] osd - peering_event(id=518, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:16.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,303 [shard 0] osd - peering_event(id=519, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:16.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:16.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:16.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - peering_event(id=519, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:16.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:16.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:16.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:54:16.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 278.937438965s (sent 278.937438965s) 2022-03-29T17:54:16.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - log is not dirty 2022-03-29T17:54:16.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:16.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:16.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:54:16.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 278.937622070s (sent 278.937622070s) 2022-03-29T17:54:16.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - log is not dirty 2022-03-29T17:54:16.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - peering_event(id=519, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:16.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:16.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:54:16.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 278.937805176s (sent 278.937805176s) 2022-03-29T17:54:16.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - log is not dirty 2022-03-29T17:54:16.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:16.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,304 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:16.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,305 [shard 0] osd - peering_event(id=521, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:16.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,306 [shard 0] osd - peering_event(id=520, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:16.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:16,306 [shard 0] osd - peering_event(id=519, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:16.738 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:16,737 [shard 0] alienstore - stat 2022-03-29T17:54:16.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~128s 2022-03-29T17:54:16.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~101s 2022-03-29T17:54:17.745 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:17,744 [shard 0] alienstore - stat 2022-03-29T17:54:17.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:17,866 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:54:17.869 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:17,867 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:54:17.869 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:17,867 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:17.869 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:17,867 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:54:17.869 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:17,867 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:54:17.927 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,925 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:17.927 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,925 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:17.927 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,925 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:54:17.928 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,925 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:54:17.928 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,925 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 280.558502197s) 2022-03-29T17:54:17.928 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,925 [shard 0] osd - log is not dirty 2022-03-29T17:54:17.928 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,925 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:17.928 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,925 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:17.929 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,926 [shard 0] osd - peering_event(id=522, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:17.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,935 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:17.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,935 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:17.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,935 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:54:17.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,935 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:54:17.938 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,935 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 280.568298340s) 2022-03-29T17:54:17.938 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,935 [shard 0] osd - log is not dirty 2022-03-29T17:54:17.938 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,935 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:17.938 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,935 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:17.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:17,936 [shard 0] osd - peering_event(id=523, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:18.391 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:18,388 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:18.391 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:18,388 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:18.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,388 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:18.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,388 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=12712478863215971039 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:18.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,388 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:18.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,388 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=3760037491441068666 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:18.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,388 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:18.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:18.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,389 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:18.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:18,389 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:18.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:18.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:18,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:18.591 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,589 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:18.591 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,589 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:18.592 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,589 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:18.592 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:18,590 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:18.592 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,590 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:18.592 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:18,590 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:18.752 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,750 [shard 0] alienstore - stat 2022-03-29T17:54:18.992 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,990 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:18.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,990 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:18.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,990 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:18.993 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:18,990 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:18.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:18,990 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:18.994 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:18,990 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:19.758 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:19,757 [shard 0] alienstore - stat 2022-03-29T17:54:19.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:19,790 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:19.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:19,790 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:19.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:19,791 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:19.794 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:19,791 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:19.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:19,791 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:19.794 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:19,791 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:20.765 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:20,764 [shard 0] alienstore - stat 2022-03-29T17:54:21.223 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:21.220+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:21.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:21,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:21.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:21,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:21.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:21,392 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:21.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:21,392 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:21.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:21,392 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:21.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:21,392 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:21.772 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:21,770 [shard 0] alienstore - stat 2022-03-29T17:54:22.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2022-03-29T17:54:22.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~106s 2022-03-29T17:54:22.779 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:22,777 [shard 0] alienstore - stat 2022-03-29T17:54:23.545 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,542 [shard 0] osd - peering_event(id=524, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:23.546 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,542 [shard 0] osd - peering_event(id=524, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:23.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,542 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:54:23.547 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,543 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:54:23.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,543 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 286.176025391s) 2022-03-29T17:54:23.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,543 [shard 0] osd - log is not dirty 2022-03-29T17:54:23.548 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,543 [shard 0] osd - peering_event(id=524, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:23.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,543 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:23.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,544 [shard 0] osd - peering_event(id=524, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:23.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,552 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:23.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,552 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:23.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,552 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:54:23.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,552 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:54:23.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,552 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 286.185852051s) 2022-03-29T17:54:23.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,552 [shard 0] osd - log is not dirty 2022-03-29T17:54:23.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,552 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:23.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,552 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:23.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,553 [shard 0] osd - peering_event(id=525, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:23.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,582 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:23.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,582 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:23.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,583 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:54:23.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,583 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:54:23.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,583 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 286.216125488s) 2022-03-29T17:54:23.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,583 [shard 0] osd - log is not dirty 2022-03-29T17:54:23.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,583 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:23.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,583 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:23.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,583 [shard 0] osd - peering_event(id=526, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:23.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,613 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:23.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,613 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:23.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,613 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:54:23.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,613 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:54:23.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,613 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 286.246246338s) 2022-03-29T17:54:23.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,613 [shard 0] osd - log is not dirty 2022-03-29T17:54:23.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,613 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:23.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,613 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:23.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,614 [shard 0] osd - peering_event(id=527, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:23.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,623 [shard 0] osd - peering_event(id=528, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:23.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,623 [shard 0] osd - peering_event(id=528, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:23.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,623 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:54:23.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,623 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:54:23.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,623 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 286.256408691s) 2022-03-29T17:54:23.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,623 [shard 0] osd - log is not dirty 2022-03-29T17:54:23.626 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,623 [shard 0] osd - peering_event(id=528, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:23.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,623 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:23.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,624 [shard 0] osd - peering_event(id=528, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:23.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,653 [shard 0] osd - peering_event(id=529, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:23.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,653 [shard 0] osd - peering_event(id=529, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:23.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,653 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:54:23.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,653 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:54:23.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,653 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 286.286621094s) 2022-03-29T17:54:23.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,653 [shard 0] osd - log is not dirty 2022-03-29T17:54:23.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,653 [shard 0] osd - peering_event(id=529, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:23.658 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,653 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:23.658 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:23,654 [shard 0] osd - peering_event(id=529, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:23.785 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:23,784 [shard 0] alienstore - stat 2022-03-29T17:54:24.294 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,287 [shard 0] osd - peering_event(id=530, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:24.294 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,287 [shard 0] osd - peering_event(id=531, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:24.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,287 [shard 0] osd - peering_event(id=532, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:24.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,287 [shard 0] osd - peering_event(id=531, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:24.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,287 [shard 0] osd - peering_event(id=532, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:24.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,287 [shard 0] osd - peering_event(id=530, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:24.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:54:24.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 286.921203613s (sent 286.921203613s) 2022-03-29T17:54:24.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - log is not dirty 2022-03-29T17:54:24.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - peering_event(id=532, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:24.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:24.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:54:24.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 286.921447754s (sent 286.921447754s) 2022-03-29T17:54:24.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - log is not dirty 2022-03-29T17:54:24.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - peering_event(id=531, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:24.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:24.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:54:24.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 286.921691895s (sent 286.921691895s) 2022-03-29T17:54:24.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - log is not dirty 2022-03-29T17:54:24.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - peering_event(id=530, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:24.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,288 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:24.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,289 [shard 0] osd - peering_event(id=532, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:24.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,289 [shard 0] osd - peering_event(id=531, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:24.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,289 [shard 0] osd - peering_event(id=530, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:24.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,307 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:24.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,307 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:24.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,307 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:24.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,307 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:24.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,307 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:24.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,307 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:24.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:54:24.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 286.941070557s (sent 286.941070557s) 2022-03-29T17:54:24.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - log is not dirty 2022-03-29T17:54:24.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:24.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:24.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:54:24.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 286.941314697s (sent 286.941314697s) 2022-03-29T17:54:24.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - log is not dirty 2022-03-29T17:54:24.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:24.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:24.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:54:24.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 286.941497803s (sent 286.941497803s) 2022-03-29T17:54:24.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - log is not dirty 2022-03-29T17:54:24.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:24.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:24.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,308 [shard 0] osd - peering_event(id=535, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:24.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,309 [shard 0] osd - peering_event(id=533, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:24.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:24,309 [shard 0] osd - peering_event(id=534, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:24.390 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:24,388 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:24.391 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:24,388 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:24.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,388 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:24.391 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,388 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=5753749574447631179 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:24.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,388 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:24.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,388 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=6925088752152573566 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:24.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,388 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:24.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:24.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:24.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:24,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:24.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,389 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:24.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:24,389 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:24.591 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,589 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:24.592 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,589 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:24.592 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,590 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:24.592 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:24,590 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:24.592 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,590 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:24.593 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:24,590 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:24.792 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,791 [shard 0] alienstore - stat 2022-03-29T17:54:24.992 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,990 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:24.992 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,990 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:24.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,990 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:24.993 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:24,991 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:24.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:24,991 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:24.994 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:24,991 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:25.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:25,791 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:25.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:25,791 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:25.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:25,791 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:25.795 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:25,792 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:25.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:25,792 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:25.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:25,792 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:25.798 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:25,797 [shard 0] alienstore - stat 2022-03-29T17:54:25.930 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,928 [shard 0] osd - peering_event(id=536, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:25.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,928 [shard 0] osd - peering_event(id=536, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:25.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,928 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:54:25.931 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,928 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:54:25.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,928 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 288.561737061s) 2022-03-29T17:54:25.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,928 [shard 0] osd - log is not dirty 2022-03-29T17:54:25.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,928 [shard 0] osd - peering_event(id=536, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:25.932 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,928 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:25.933 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,929 [shard 0] osd - peering_event(id=536, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:25.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,938 [shard 0] osd - peering_event(id=537, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:25.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,938 [shard 0] osd - peering_event(id=537, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:25.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,938 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:54:25.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,938 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:54:25.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,938 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 288.571838379s) 2022-03-29T17:54:25.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,938 [shard 0] osd - log is not dirty 2022-03-29T17:54:25.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,938 [shard 0] osd - peering_event(id=537, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:25.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,938 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:25.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:25,939 [shard 0] osd - peering_event(id=537, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:26.224 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:26.221+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:26.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:26,804 [shard 0] alienstore - stat 2022-03-29T17:54:27.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:27,392 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:27.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:27,392 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:27.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:27,393 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:27.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:27,393 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:27.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:27,393 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:27.396 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:27,393 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:27.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2022-03-29T17:54:27.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~111s 2022-03-29T17:54:27.813 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:27,811 [shard 0] alienstore - stat 2022-03-29T17:54:28.820 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:28,818 [shard 0] alienstore - stat 2022-03-29T17:54:29.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:29,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:29.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:29,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:54:29.285 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:29,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:29.285 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:29,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:54:29.285 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:29,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:54:29.827 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:29,825 [shard 0] alienstore - stat 2022-03-29T17:54:30.391 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:30,389 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:30.392 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:30,389 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:30.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:30.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=4217863826014321470 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:30.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,389 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:30.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,389 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=9670931921239423142 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:30.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,389 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:30.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:30.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:30.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:30,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:30.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:30.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:30,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:30.592 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,590 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:30.592 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,590 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:30.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:30.593 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:30,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:30.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:30.594 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:30,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:30.833 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,832 [shard 0] alienstore - stat 2022-03-29T17:54:30.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,991 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:30.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,991 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:30.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,991 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:30.994 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:30,991 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:30.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:30,991 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:30.994 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:30,991 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:31.225 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:31.222+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,546 [shard 0] osd - peering_event(id=538, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,546 [shard 0] osd - peering_event(id=538, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,547 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:54:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,547 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:54:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,547 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 294.180206299s) 2022-03-29T17:54:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,547 [shard 0] osd - log is not dirty 2022-03-29T17:54:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,547 [shard 0] osd - peering_event(id=538, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:31.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,547 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:31.551 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,548 [shard 0] osd - peering_event(id=538, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:31.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,556 [shard 0] osd - peering_event(id=539, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:31.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,557 [shard 0] osd - peering_event(id=539, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:31.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,557 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:54:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,557 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:54:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,557 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 294.190277100s) 2022-03-29T17:54:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,557 [shard 0] osd - log is not dirty 2022-03-29T17:54:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,557 [shard 0] osd - peering_event(id=539, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:31.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,557 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:31.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,558 [shard 0] osd - peering_event(id=539, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:31.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,587 [shard 0] osd - peering_event(id=540, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:31.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,587 [shard 0] osd - peering_event(id=540, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:31.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,587 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:54:31.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,587 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:54:31.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,587 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 294.220489502s) 2022-03-29T17:54:31.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,587 [shard 0] osd - log is not dirty 2022-03-29T17:54:31.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,587 [shard 0] osd - peering_event(id=540, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:31.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,587 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:31.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,588 [shard 0] osd - peering_event(id=540, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:31.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,617 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,617 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,617 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:54:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,617 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:54:31.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,617 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 294.250732422s) 2022-03-29T17:54:31.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,617 [shard 0] osd - log is not dirty 2022-03-29T17:54:31.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,617 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:31.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,617 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:31.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,618 [shard 0] osd - peering_event(id=541, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:31.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,627 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:31.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,627 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:31.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,627 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:54:31.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,627 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:54:31.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,627 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 294.260742188s) 2022-03-29T17:54:31.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,627 [shard 0] osd - log is not dirty 2022-03-29T17:54:31.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,627 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:31.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,627 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:31.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,628 [shard 0] osd - peering_event(id=542, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:31.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,657 [shard 0] osd - peering_event(id=543, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:31.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,657 [shard 0] osd - peering_event(id=543, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:31.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,657 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:54:31.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,657 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:54:31.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,657 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 294.290863037s) 2022-03-29T17:54:31.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,657 [shard 0] osd - log is not dirty 2022-03-29T17:54:31.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,657 [shard 0] osd - peering_event(id=543, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:31.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,657 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:31.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:31,658 [shard 0] osd - peering_event(id=543, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:31.793 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:31,792 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:31.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:31,792 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:31.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:31,792 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:31.795 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:31,792 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:31.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:31,792 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:31.795 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:31,792 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:31.840 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:31,838 [shard 0] alienstore - stat 2022-03-29T17:54:32.295 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - peering_event(id=544, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:32.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - peering_event(id=545, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:32.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - peering_event(id=546, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:32.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - peering_event(id=545, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:32.296 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - peering_event(id=546, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:32.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - peering_event(id=544, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:32.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:54:32.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 294.925689697s (sent 294.925689697s) 2022-03-29T17:54:32.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - log is not dirty 2022-03-29T17:54:32.297 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - peering_event(id=544, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:32.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:32.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:54:32.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 294.925964355s (sent 294.925964355s) 2022-03-29T17:54:32.298 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - log is not dirty 2022-03-29T17:54:32.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,292 [shard 0] osd - peering_event(id=545, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:32.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:32.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:54:32.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 294.926147461s (sent 294.926147461s) 2022-03-29T17:54:32.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - log is not dirty 2022-03-29T17:54:32.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - peering_event(id=546, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:32.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:32.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - peering_event(id=544, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:32.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - peering_event(id=546, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:32.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,293 [shard 0] osd - peering_event(id=545, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:32.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - peering_event(id=547, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:32.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:32.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:32.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - peering_event(id=547, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:32.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:32.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:32.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:54:32.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 294.945678711s (sent 294.945678711s) 2022-03-29T17:54:32.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - log is not dirty 2022-03-29T17:54:32.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:32.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:32.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:54:32.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 294.945861816s (sent 294.945861816s) 2022-03-29T17:54:32.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - log is not dirty 2022-03-29T17:54:32.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - peering_event(id=547, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:32.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:32.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,312 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:54:32.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,313 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 294.946044922s (sent 294.946044922s) 2022-03-29T17:54:32.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,313 [shard 0] osd - log is not dirty 2022-03-29T17:54:32.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,313 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:32.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,313 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:32.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,313 [shard 0] osd - peering_event(id=547, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:32.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,313 [shard 0] osd - peering_event(id=549, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:32.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:32,316 [shard 0] osd - peering_event(id=548, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:32.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~144s 2022-03-29T17:54:32.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~117s 2022-03-29T17:54:32.847 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:32,846 [shard 0] alienstore - stat 2022-03-29T17:54:32.869 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:32,867 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:54:32.869 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:32,868 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:54:32.870 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:32,868 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:32.870 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:32,868 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:54:32.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:32,868 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:54:33.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:33,392 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:33.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:33,393 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:33.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:33,393 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:33.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:33,393 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:33.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:33,393 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:33.396 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:33,393 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:33.854 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:33,852 [shard 0] alienstore - stat 2022-03-29T17:54:33.935 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,933 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:33.935 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,933 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:33.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,933 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:54:33.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,933 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:54:33.936 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,933 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 296.566680908s) 2022-03-29T17:54:33.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,933 [shard 0] osd - log is not dirty 2022-03-29T17:54:33.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,933 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:33.937 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,933 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:33.938 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,934 [shard 0] osd - peering_event(id=550, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:33.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,943 [shard 0] osd - peering_event(id=551, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:33.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,943 [shard 0] osd - peering_event(id=551, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:33.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,943 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:54:33.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,943 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:54:33.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,943 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 296.576721191s) 2022-03-29T17:54:33.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,943 [shard 0] osd - log is not dirty 2022-03-29T17:54:33.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,943 [shard 0] osd - peering_event(id=551, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:33.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,943 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:33.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:33,944 [shard 0] osd - peering_event(id=551, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:34.861 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:34,859 [shard 0] alienstore - stat 2022-03-29T17:54:35.868 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:35,866 [shard 0] alienstore - stat 2022-03-29T17:54:36.226 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:36.224+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:36.392 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:36,389 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:36.392 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:36,389 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:36.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:36.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=1629297113253472291 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:36.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:36.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=15069709901476548642 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:36.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:36.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:36.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:36.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:36,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:36.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:36.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:36,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:36.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:36.595 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:36.595 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:36.596 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:36,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:36.596 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,592 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:36.596 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:36,592 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:36.874 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,873 [shard 0] alienstore - stat 2022-03-29T17:54:36.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:36.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:36.996 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:36.996 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:36,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:36.997 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:36,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:36.997 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:36,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:37.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:37,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:37.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:37,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:37.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:37,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:37.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:37,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:37.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:37,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:37.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:37,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:37.881 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:37,879 [shard 0] alienstore - stat 2022-03-29T17:54:38.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~149s 2022-03-29T17:54:38.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~122s 2022-03-29T17:54:38.888 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:38,886 [shard 0] alienstore - stat 2022-03-29T17:54:39.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:39,393 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:39.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:39,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:39.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:39,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:39.396 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:39,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:39.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:39,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:39.397 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:39,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:39.553 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,550 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:39.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,550 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:39.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,551 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:54:39.554 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,551 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:54:39.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,551 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 302.184204102s) 2022-03-29T17:54:39.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,551 [shard 0] osd - log is not dirty 2022-03-29T17:54:39.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,551 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:39.555 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,551 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:39.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,552 [shard 0] osd - peering_event(id=552, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:39.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,560 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:39.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,560 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:39.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,561 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:54:39.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,561 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:54:39.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,561 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 302.194274902s) 2022-03-29T17:54:39.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,561 [shard 0] osd - log is not dirty 2022-03-29T17:54:39.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,561 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:39.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,561 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:39.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,561 [shard 0] osd - peering_event(id=553, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:39.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,591 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:39.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,591 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:39.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,591 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:54:39.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,591 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:54:39.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,591 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 302.224304199s) 2022-03-29T17:54:39.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,591 [shard 0] osd - log is not dirty 2022-03-29T17:54:39.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,591 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:39.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,591 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:39.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,592 [shard 0] osd - peering_event(id=554, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:39.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,621 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:39.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,621 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:39.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,621 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:54:39.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,621 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:54:39.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,621 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 302.254577637s) 2022-03-29T17:54:39.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,621 [shard 0] osd - log is not dirty 2022-03-29T17:54:39.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,621 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:39.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,621 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:39.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,622 [shard 0] osd - peering_event(id=555, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:39.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,631 [shard 0] osd - peering_event(id=556, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:39.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,631 [shard 0] osd - peering_event(id=556, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:39.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,631 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:54:39.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,631 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:54:39.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,631 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 302.264709473s) 2022-03-29T17:54:39.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,631 [shard 0] osd - log is not dirty 2022-03-29T17:54:39.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,631 [shard 0] osd - peering_event(id=556, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:39.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,631 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:39.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,632 [shard 0] osd - peering_event(id=556, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:39.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,661 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:39.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,661 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:39.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,661 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:54:39.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,661 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:54:39.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,661 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 302.294952393s) 2022-03-29T17:54:39.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,662 [shard 0] osd - log is not dirty 2022-03-29T17:54:39.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,662 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:39.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,662 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:39.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:39,662 [shard 0] osd - peering_event(id=557, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:39.894 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:39,893 [shard 0] alienstore - stat 2022-03-29T17:54:40.299 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,295 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:40.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:40.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - peering_event(id=560, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:40.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - peering_event(id=560, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:40.300 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:40.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:40.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:54:40.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 302.929504395s (sent 302.929504395s) 2022-03-29T17:54:40.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - log is not dirty 2022-03-29T17:54:40.301 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:40.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:40.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:54:40.302 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 302.929748535s (sent 302.929748535s) 2022-03-29T17:54:40.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - log is not dirty 2022-03-29T17:54:40.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - peering_event(id=560, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:40.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:40.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:54:40.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 302.929931641s (sent 302.929931641s) 2022-03-29T17:54:40.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - log is not dirty 2022-03-29T17:54:40.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:40.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,296 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:40.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,297 [shard 0] osd - peering_event(id=559, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:40.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,297 [shard 0] osd - peering_event(id=560, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:40.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,297 [shard 0] osd - peering_event(id=558, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:40.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - peering_event(id=561, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:40.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - peering_event(id=562, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:40.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - peering_event(id=563, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:40.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - peering_event(id=563, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:40.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - peering_event(id=562, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:40.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - peering_event(id=561, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:40.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:54:40.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 302.949676514s (sent 302.949676514s) 2022-03-29T17:54:40.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - log is not dirty 2022-03-29T17:54:40.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - peering_event(id=563, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:40.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:40.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:54:40.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 302.949859619s (sent 302.949859619s) 2022-03-29T17:54:40.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - log is not dirty 2022-03-29T17:54:40.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - peering_event(id=562, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:40.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,316 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:40.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,317 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:54:40.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,317 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 302.950103760s (sent 302.950103760s) 2022-03-29T17:54:40.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,317 [shard 0] osd - log is not dirty 2022-03-29T17:54:40.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,317 [shard 0] osd - peering_event(id=561, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:40.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:40.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,317 [shard 0] osd - peering_event(id=563, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:40.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,318 [shard 0] osd - peering_event(id=561, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:40.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:40,318 [shard 0] osd - peering_event(id=562, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:40.901 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:40,900 [shard 0] alienstore - stat 2022-03-29T17:54:41.227 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:41.225+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:41.908 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:41,906 [shard 0] alienstore - stat 2022-03-29T17:54:41.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,936 [shard 0] osd - peering_event(id=564, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:41.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,936 [shard 0] osd - peering_event(id=564, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:41.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,936 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:54:41.939 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,937 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:54:41.940 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,937 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 304.570037842s) 2022-03-29T17:54:41.941 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,937 [shard 0] osd - log is not dirty 2022-03-29T17:54:41.941 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,937 [shard 0] osd - peering_event(id=564, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:41.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,937 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:41.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,938 [shard 0] osd - peering_event(id=564, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:41.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,946 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:41.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,946 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:41.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,946 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:54:41.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,947 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:54:41.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,947 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 304.580047607s) 2022-03-29T17:54:41.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,947 [shard 0] osd - log is not dirty 2022-03-29T17:54:41.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,947 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:41.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,947 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:41.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:41,947 [shard 0] osd - peering_event(id=565, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:42.392 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:42,389 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:42.392 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:42,389 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:42.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:42.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=10565190271741654804 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:42.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:42.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=10384179101385039389 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:42.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:42.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:42.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:42.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:42,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:42.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:42.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:42,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:42.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:42.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:42.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:42.594 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:42,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:42.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:42.595 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:42,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:42.914 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,913 [shard 0] alienstore - stat 2022-03-29T17:54:42.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:42.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:42.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:42.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:42,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:42.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:42,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:42.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:42,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:43.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2022-03-29T17:54:43.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~127s 2022-03-29T17:54:43.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:43,792 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:43.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:43,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:43.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:43,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:43.797 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:43,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:43.797 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:43,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:43.798 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:43,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:43.921 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:43,920 [shard 0] alienstore - stat 2022-03-29T17:54:44.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:44,282 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:44.285 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:44,283 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:54:44.285 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:44,283 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:44.285 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:44,283 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:54:44.285 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:44,283 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:54:44.928 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:44,927 [shard 0] alienstore - stat 2022-03-29T17:54:45.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:45,393 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:45.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:45,393 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:45.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:45,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:45.396 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:45,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:45.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:45,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:45.397 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:45,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:45.935 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:45,934 [shard 0] alienstore - stat 2022-03-29T17:54:46.228 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:46.225+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:46.942 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:46,940 [shard 0] alienstore - stat 2022-03-29T17:54:47.556 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,554 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:47.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,554 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:47.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,554 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:54:47.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,555 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:54:47.557 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,555 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 310.188018799s) 2022-03-29T17:54:47.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,555 [shard 0] osd - log is not dirty 2022-03-29T17:54:47.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,555 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:47.558 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,555 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:47.559 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,556 [shard 0] osd - peering_event(id=566, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:47.566 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,564 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:47.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,564 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:47.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,564 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:54:47.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,564 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:54:47.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,564 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 310.197937012s) 2022-03-29T17:54:47.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,565 [shard 0] osd - log is not dirty 2022-03-29T17:54:47.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,565 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:47.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,565 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:47.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,565 [shard 0] osd - peering_event(id=567, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:47.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,594 [shard 0] osd - peering_event(id=568, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:47.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,594 [shard 0] osd - peering_event(id=568, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:47.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,594 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:54:47.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,595 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:54:47.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,595 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 310.227996826s) 2022-03-29T17:54:47.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,595 [shard 0] osd - log is not dirty 2022-03-29T17:54:47.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,595 [shard 0] osd - peering_event(id=568, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:47.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,595 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:47.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,595 [shard 0] osd - peering_event(id=568, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:47.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - peering_event(id=569, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:47.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - peering_event(id=569, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:47.627 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:54:47.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:54:47.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 310.258239746s) 2022-03-29T17:54:47.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - log is not dirty 2022-03-29T17:54:47.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - peering_event(id=569, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:47.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:47.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,625 [shard 0] osd - peering_event(id=569, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:47.637 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - peering_event(id=570, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:47.637 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - peering_event(id=570, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:47.637 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:54:47.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:54:47.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 310.268371582s) 2022-03-29T17:54:47.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - log is not dirty 2022-03-29T17:54:47.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - peering_event(id=570, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:47.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:47.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,635 [shard 0] osd - peering_event(id=570, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:47.667 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,665 [shard 0] osd - peering_event(id=571, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:47.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,665 [shard 0] osd - peering_event(id=571, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:47.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,665 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:54:47.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,665 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:54:47.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,665 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 310.298736572s) 2022-03-29T17:54:47.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,665 [shard 0] osd - log is not dirty 2022-03-29T17:54:47.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,665 [shard 0] osd - peering_event(id=571, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:47.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,665 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:47.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:47,666 [shard 0] osd - peering_event(id=571, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:47.870 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:47,868 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:54:47.870 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:47,868 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:54:47.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:47,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:47.871 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:47,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:54:47.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:47,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:54:47.949 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:47,947 [shard 0] alienstore - stat 2022-03-29T17:54:48.303 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,299 [shard 0] osd - peering_event(id=572, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:48.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,299 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:48.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,299 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:48.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,299 [shard 0] osd - peering_event(id=572, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:48.304 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,299 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:48.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:48.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:54:48.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 310.933258057s (sent 310.933258057s) 2022-03-29T17:54:48.305 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - log is not dirty 2022-03-29T17:54:48.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:48.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:48.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:54:48.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 310.933502197s (sent 310.933502197s) 2022-03-29T17:54:48.306 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - log is not dirty 2022-03-29T17:54:48.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - peering_event(id=572, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:48.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:48.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:54:48.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 310.933624268s (sent 310.933624268s) 2022-03-29T17:54:48.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - log is not dirty 2022-03-29T17:54:48.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:48.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:48.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,300 [shard 0] osd - peering_event(id=573, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:48.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,301 [shard 0] osd - peering_event(id=574, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:48.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,301 [shard 0] osd - peering_event(id=572, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:48.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,319 [shard 0] osd - peering_event(id=575, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:48.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,319 [shard 0] osd - peering_event(id=576, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:48.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,319 [shard 0] osd - peering_event(id=577, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:48.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,319 [shard 0] osd - peering_event(id=577, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:48.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,319 [shard 0] osd - peering_event(id=576, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:48.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - peering_event(id=575, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:48.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:54:48.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 310.953247070s (sent 310.953247070s) 2022-03-29T17:54:48.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - log is not dirty 2022-03-29T17:54:48.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - peering_event(id=577, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:48.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:48.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:54:48.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 310.953552246s (sent 310.953552246s) 2022-03-29T17:54:48.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - log is not dirty 2022-03-29T17:54:48.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - peering_event(id=576, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:48.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:48.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:54:48.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 310.953735352s (sent 310.953735352s) 2022-03-29T17:54:48.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - log is not dirty 2022-03-29T17:54:48.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - peering_event(id=575, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:48.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,320 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:48.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,321 [shard 0] osd - peering_event(id=577, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:48.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,321 [shard 0] osd - peering_event(id=576, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:48.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:48,321 [shard 0] osd - peering_event(id=575, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:48.392 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:48,390 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:48.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:48,390 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:48.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:48.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=9142434127766674218 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:48.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:48.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=14561717801698107724 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:48.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:48.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:48.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:48.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:48,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:48.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:48.396 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:48,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:48.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:48.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:48.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:48.595 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:48,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:48.595 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,592 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:48.595 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:48,592 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:48.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2022-03-29T17:54:48.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~133s 2022-03-29T17:54:48.955 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,954 [shard 0] alienstore - stat 2022-03-29T17:54:48.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:48.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:48.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:48.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:48,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:48.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:48,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:48.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:48,993 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:49.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:49,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:49.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:49,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:49.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:49,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:49.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:49,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:49.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:49,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:49.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:49,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:49.942 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,940 [shard 0] osd - peering_event(id=578, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:49.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,940 [shard 0] osd - peering_event(id=578, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:49.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,940 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:54:49.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,940 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:54:49.943 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,940 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 312.573638916s) 2022-03-29T17:54:49.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,940 [shard 0] osd - log is not dirty 2022-03-29T17:54:49.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,940 [shard 0] osd - peering_event(id=578, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:49.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,940 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:49.944 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,941 [shard 0] osd - peering_event(id=578, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:49.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,950 [shard 0] osd - peering_event(id=579, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:49.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,950 [shard 0] osd - peering_event(id=579, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:49.952 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,950 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:54:49.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,950 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:54:49.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,950 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 312.583648682s) 2022-03-29T17:54:49.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,950 [shard 0] osd - log is not dirty 2022-03-29T17:54:49.953 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,950 [shard 0] osd - peering_event(id=579, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:49.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,950 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:49.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:49,951 [shard 0] osd - peering_event(id=579, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:49.961 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:49,960 [shard 0] alienstore - stat 2022-03-29T17:54:50.968 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:50,967 [shard 0] alienstore - stat 2022-03-29T17:54:51.229 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:51.226+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:51.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:51,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:51.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:51,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:51.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:51,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:51.399 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:51,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:51.399 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:51,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:51.399 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:51,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:51.975 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:51,973 [shard 0] alienstore - stat 2022-03-29T17:54:52.981 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:52,980 [shard 0] alienstore - stat 2022-03-29T17:54:53.988 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:53,987 [shard 0] alienstore - stat 2022-03-29T17:54:54.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2022-03-29T17:54:54.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~138s 2022-03-29T17:54:54.392 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:54:54,389 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:54:54.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:54,389 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:54:54.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:54:54.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=2009421323701273439 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:54.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:54:54.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=4486409425551065427 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:54:54.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:54.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:54:54.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:54.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:54,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:54:54.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:54.396 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:54,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:54:54.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:54.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:54.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:54.594 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:54,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:54:54.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:54.594 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:54,592 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:54:54.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:54.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:54.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:54.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:54,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:54:54.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:54.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:54,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:54:54.996 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:54,994 [shard 0] alienstore - stat 2022-03-29T17:54:55.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,557 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:55.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,557 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:55.560 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,558 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:54:55.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,558 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:54:55.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,558 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 318.191070557s) 2022-03-29T17:54:55.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,558 [shard 0] osd - log is not dirty 2022-03-29T17:54:55.561 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,558 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:55.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,558 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:55.562 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,559 [shard 0] osd - peering_event(id=580, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:55.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,567 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:55.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,567 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:55.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,568 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:54:55.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,568 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:54:55.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,568 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 318.201141357s) 2022-03-29T17:54:55.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,568 [shard 0] osd - log is not dirty 2022-03-29T17:54:55.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,568 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:55.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,568 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:55.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,568 [shard 0] osd - peering_event(id=581, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:55.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,598 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:55.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,598 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:55.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,598 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:54:55.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,598 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:54:55.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,598 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 318.231384277s) 2022-03-29T17:54:55.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,598 [shard 0] osd - log is not dirty 2022-03-29T17:54:55.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,598 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:55.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,598 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:55.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,599 [shard 0] osd - peering_event(id=582, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:55.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,628 [shard 0] osd - peering_event(id=583, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:55.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,628 [shard 0] osd - peering_event(id=583, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:55.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,628 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:54:55.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,628 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:54:55.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,628 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 318.261566162s) 2022-03-29T17:54:55.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,628 [shard 0] osd - log is not dirty 2022-03-29T17:54:55.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,628 [shard 0] osd - peering_event(id=583, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:55.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,628 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:55.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,629 [shard 0] osd - peering_event(id=583, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:55.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,638 [shard 0] osd - peering_event(id=584, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:55.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,638 [shard 0] osd - peering_event(id=584, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:55.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,638 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:54:55.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,638 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:54:55.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,638 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 318.271636963s) 2022-03-29T17:54:55.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,638 [shard 0] osd - log is not dirty 2022-03-29T17:54:55.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,638 [shard 0] osd - peering_event(id=584, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:55.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,638 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:55.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,639 [shard 0] osd - peering_event(id=584, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:55.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,668 [shard 0] osd - peering_event(id=585, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:54:55.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,668 [shard 0] osd - peering_event(id=585, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:54:55.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,668 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:54:55.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,668 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:54:55.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,668 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 318.301910400s) 2022-03-29T17:54:55.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,668 [shard 0] osd - log is not dirty 2022-03-29T17:54:55.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,668 [shard 0] osd - peering_event(id=585, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:54:55.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,668 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:55.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:55,669 [shard 0] osd - peering_event(id=585, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:54:55.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:55,792 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:55.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:55,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:55.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:55,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:55.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:55,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:54:55.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:55,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:55.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:55,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:54:56.002 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:56,000 [shard 0] alienstore - stat 2022-03-29T17:54:56.230 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:54:56.227+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:54:56.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,302 [shard 0] osd - peering_event(id=586, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:56.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,302 [shard 0] osd - peering_event(id=587, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:56.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - peering_event(id=588, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:56.307 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - peering_event(id=588, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:56.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - peering_event(id=587, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:56.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - peering_event(id=586, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:56.308 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:54:56.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 318.936462402s (sent 318.936462402s) 2022-03-29T17:54:56.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - log is not dirty 2022-03-29T17:54:56.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - peering_event(id=588, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:56.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:56.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:54:56.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 318.936737061s (sent 318.936737061s) 2022-03-29T17:54:56.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - log is not dirty 2022-03-29T17:54:56.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - peering_event(id=587, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:56.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:56.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,303 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:54:56.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,304 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 318.937011719s (sent 318.937011719s) 2022-03-29T17:54:56.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,304 [shard 0] osd - log is not dirty 2022-03-29T17:54:56.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,304 [shard 0] osd - peering_event(id=586, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:56.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,304 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:56.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,304 [shard 0] osd - peering_event(id=588, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:56.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,304 [shard 0] osd - peering_event(id=587, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:56.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,304 [shard 0] osd - peering_event(id=586, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:56.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,322 [shard 0] osd - peering_event(id=589, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:56.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,322 [shard 0] osd - peering_event(id=590, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:56.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,322 [shard 0] osd - peering_event(id=591, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:54:56.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,322 [shard 0] osd - peering_event(id=590, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:56.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,322 [shard 0] osd - peering_event(id=589, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:56.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - peering_event(id=591, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:54:56.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:54:56.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 318.956237793s (sent 318.956237793s) 2022-03-29T17:54:56.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - log is not dirty 2022-03-29T17:54:56.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - peering_event(id=589, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:56.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:56.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:54:56.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 318.956451416s (sent 318.956451416s) 2022-03-29T17:54:56.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - log is not dirty 2022-03-29T17:54:56.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - peering_event(id=590, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:56.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:56.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:54:56.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 318.956634521s (sent 318.956634521s) 2022-03-29T17:54:56.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - log is not dirty 2022-03-29T17:54:56.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - peering_event(id=591, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:54:56.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,323 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:56.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,324 [shard 0] osd - peering_event(id=590, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:56.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,324 [shard 0] osd - peering_event(id=589, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:56.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,324 [shard 0] osd - peering_event(id=591, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:54:56.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,756 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4127.0:10 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): start 2022-03-29T17:54:56.763 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,756 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4127.0:10 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): in repeat 2022-03-29T17:54:56.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,756 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4127.0:10 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]) same_interval_since: 11 2022-03-29T17:54:56.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,756 [shard 0] osd - OpSequencer::start_op: op=11, last_started=9, last_unblocked=9, last_completed=9 2022-03-29T17:54:56.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - do_recover_missing check for recovery, 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:54:56.764 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4127.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): got obc lock 2022-03-29T17:54:56.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - with_head_obc 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:54:56.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - with_head_obc: found 2:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:54:56.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:54:56.765 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - handling op omap-get-vals on object 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:54:56.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:54:56.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8 - object 2:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:54:56.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,757 [shard 0] osd - with_head_obc: released 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:54:56.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,758 [shard 0] osd - OpSequencer::finish_op_in_order: op=11, last_started=11, last_unblocked=11, last_completed=9 2022-03-29T17:54:56.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,758 [shard 0] osd - client_request(id=11, detail=m=[osd_op(client.4127.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): destroying 2022-03-29T17:54:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,758 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4127.0:11 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): start 2022-03-29T17:54:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,758 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4127.0:11 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): in repeat 2022-03-29T17:54:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,758 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4127.0:11 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]) same_interval_since: 14 2022-03-29T17:54:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,758 [shard 0] osd - OpSequencer::start_op: op=12, last_started=10, last_unblocked=10, last_completed=10 2022-03-29T17:54:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,758 [shard 0] osd - do_recover_missing check for recovery, 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:54:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4127.0:11 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): got obc lock 2022-03-29T17:54:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - with_head_obc 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:54:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - with_head_obc: found 3:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:54:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:54:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - handling op omap-get-vals on object 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:54:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:54:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:11 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8 - object 3:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:54:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - with_head_obc: released 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:54:56.770 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - OpSequencer::finish_op_in_order: op=12, last_started=12, last_unblocked=12, last_completed=10 2022-03-29T17:54:56.770 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:56,759 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.4127.0:11 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): destroying 2022-03-29T17:54:57.009 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:57,007 [shard 0] alienstore - stat 2022-03-29T17:54:57.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:57,393 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:57.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:57,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:57.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:57,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:57.397 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:57,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:54:57.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:57,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:57.397 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:57,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:54:57.945 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,943 [shard 0] osd - peering_event(id=592, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:57.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,943 [shard 0] osd - peering_event(id=592, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:57.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,943 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:54:57.946 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,944 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:54:57.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,944 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 320.577026367s) 2022-03-29T17:54:57.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,944 [shard 0] osd - log is not dirty 2022-03-29T17:54:57.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,944 [shard 0] osd - peering_event(id=592, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:57.947 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,944 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:57.948 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,945 [shard 0] osd - peering_event(id=592, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:57.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,953 [shard 0] osd - peering_event(id=593, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:54:57.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,953 [shard 0] osd - peering_event(id=593, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:54:57.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,953 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:54:57.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,953 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:54:57.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,954 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 320.587005615s) 2022-03-29T17:54:57.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,954 [shard 0] osd - log is not dirty 2022-03-29T17:54:57.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,954 [shard 0] osd - peering_event(id=593, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:54:57.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,954 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:54:57.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:54:57,954 [shard 0] osd - peering_event(id=593, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:54:58.015 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:58,014 [shard 0] alienstore - stat 2022-03-29T17:54:59.022 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:59,021 [shard 0] alienstore - stat 2022-03-29T17:54:59.285 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:59,283 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:54:59.285 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:59,283 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:54:59.286 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:59,283 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:54:59.286 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:54:59,283 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:54:59.286 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:54:59,284 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:54:59.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2022-03-29T17:54:59.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~143s 2022-03-29T17:55:00.029 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,027 [shard 0] alienstore - stat 2022-03-29T17:55:00.393 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:00,389 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:00.394 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:00,389 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:00.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,389 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:00.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=15534019853452877928 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:00.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:00.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=9504572137149304535 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:00.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:00.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:00.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:00.398 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:00,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:00.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:00.399 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:00,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:00.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:00.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:00.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:00.594 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:00,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:00.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:00.594 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:00,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:00.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,991 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:00.993 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:00.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:00.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:00,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:00.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:00,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:00.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:00,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:01.035 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:01,034 [shard 0] alienstore - stat 2022-03-29T17:55:01.231 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:01.228+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:01.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:01,792 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:01.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:01,792 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:01.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:01,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:01.795 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:01,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:01.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:01,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:01.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:01,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:02.042 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:02,041 [shard 0] alienstore - stat 2022-03-29T17:55:02.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:02,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:55:02.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:02,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:55:02.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:02,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:02.872 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:02,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:55:02.872 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:02,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:55:03.049 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:03,047 [shard 0] alienstore - stat 2022-03-29T17:55:03.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:03,393 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:03.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:03,393 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:03.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:03,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:03.396 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:03,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:03.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:03,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:03.396 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:03,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:03.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,561 [shard 0] osd - peering_event(id=594, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:03.563 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,561 [shard 0] osd - peering_event(id=594, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,561 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:55:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,561 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:55:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,561 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 326.194427490s) 2022-03-29T17:55:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,561 [shard 0] osd - log is not dirty 2022-03-29T17:55:03.564 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,561 [shard 0] osd - peering_event(id=594, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:03.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,561 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:03.565 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,562 [shard 0] osd - peering_event(id=594, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:03.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,571 [shard 0] osd - peering_event(id=595, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:03.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,571 [shard 0] osd - peering_event(id=595, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:03.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,571 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:55:03.573 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,571 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:55:03.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,571 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 326.204467773s) 2022-03-29T17:55:03.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,571 [shard 0] osd - log is not dirty 2022-03-29T17:55:03.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,571 [shard 0] osd - peering_event(id=595, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:03.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,571 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:03.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,572 [shard 0] osd - peering_event(id=595, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:03.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,601 [shard 0] osd - peering_event(id=596, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:03.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,601 [shard 0] osd - peering_event(id=596, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:03.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,601 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:55:03.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,601 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:55:03.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,601 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 326.234497070s) 2022-03-29T17:55:03.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,601 [shard 0] osd - log is not dirty 2022-03-29T17:55:03.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,601 [shard 0] osd - peering_event(id=596, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:03.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,601 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:03.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,602 [shard 0] osd - peering_event(id=596, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:03.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,631 [shard 0] osd - peering_event(id=597, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:03.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,631 [shard 0] osd - peering_event(id=597, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:03.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,631 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:55:03.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,631 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:55:03.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,631 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 326.264892578s) 2022-03-29T17:55:03.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,632 [shard 0] osd - log is not dirty 2022-03-29T17:55:03.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,632 [shard 0] osd - peering_event(id=597, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:03.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,632 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:03.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,632 [shard 0] osd - peering_event(id=597, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:03.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,641 [shard 0] osd - peering_event(id=598, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:03.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,641 [shard 0] osd - peering_event(id=598, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:03.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,641 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:55:03.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,641 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:55:03.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,641 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 326.274780273s) 2022-03-29T17:55:03.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,641 [shard 0] osd - log is not dirty 2022-03-29T17:55:03.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,641 [shard 0] osd - peering_event(id=598, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:03.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,641 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:03.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,642 [shard 0] osd - peering_event(id=598, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:03.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,671 [shard 0] osd - peering_event(id=599, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:03.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,671 [shard 0] osd - peering_event(id=599, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:03.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,671 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:55:03.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,671 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:55:03.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,671 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 326.304962158s) 2022-03-29T17:55:03.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,672 [shard 0] osd - log is not dirty 2022-03-29T17:55:03.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,672 [shard 0] osd - peering_event(id=599, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:03.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,672 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:03.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:03,672 [shard 0] osd - peering_event(id=599, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:04.056 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:04,054 [shard 0] alienstore - stat 2022-03-29T17:55:04.309 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - peering_event(id=600, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:04.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - peering_event(id=601, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:04.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - peering_event(id=602, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:04.310 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - peering_event(id=601, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:04.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - peering_event(id=602, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:04.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - peering_event(id=600, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:04.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:55:04.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 326.939636230s (sent 326.939636230s) 2022-03-29T17:55:04.311 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - log is not dirty 2022-03-29T17:55:04.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - peering_event(id=600, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:04.312 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:04.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:55:04.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 326.939880371s (sent 326.939880371s) 2022-03-29T17:55:04.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - log is not dirty 2022-03-29T17:55:04.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - peering_event(id=601, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:04.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:04.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,306 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:55:04.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,307 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 326.940002441s (sent 326.940002441s) 2022-03-29T17:55:04.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,307 [shard 0] osd - log is not dirty 2022-03-29T17:55:04.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,307 [shard 0] osd - peering_event(id=602, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:04.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,307 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:04.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,307 [shard 0] osd - peering_event(id=600, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:04.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,307 [shard 0] osd - peering_event(id=602, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:04.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,307 [shard 0] osd - peering_event(id=601, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:04.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - peering_event(id=603, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:04.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:04.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:04.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:04.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - peering_event(id=603, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:04.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:04.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:55:04.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 326.959686279s (sent 326.959686279s) 2022-03-29T17:55:04.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - log is not dirty 2022-03-29T17:55:04.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - peering_event(id=603, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:04.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:04.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:55:04.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 326.959899902s (sent 326.959899902s) 2022-03-29T17:55:04.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - log is not dirty 2022-03-29T17:55:04.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:04.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:04.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,327 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:55:04.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,327 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 326.960083008s (sent 326.960083008s) 2022-03-29T17:55:04.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,327 [shard 0] osd - log is not dirty 2022-03-29T17:55:04.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,327 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:04.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,327 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:04.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,327 [shard 0] osd - peering_event(id=603, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:04.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,329 [shard 0] osd - peering_event(id=604, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:04.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:04,330 [shard 0] osd - peering_event(id=605, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:04.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~176s 2022-03-29T17:55:04.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~148s 2022-03-29T17:55:05.062 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:05,061 [shard 0] alienstore - stat 2022-03-29T17:55:05.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,947 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:05.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,947 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:05.949 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,947 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:55:05.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,947 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:55:05.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,947 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 328.580352783s) 2022-03-29T17:55:05.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,947 [shard 0] osd - log is not dirty 2022-03-29T17:55:05.950 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,947 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:05.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,947 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:05.951 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,948 [shard 0] osd - peering_event(id=606, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:05.959 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,957 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:05.959 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,957 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:05.959 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,957 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:55:05.960 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,957 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:55:05.960 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,957 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 328.590423584s) 2022-03-29T17:55:05.960 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,957 [shard 0] osd - log is not dirty 2022-03-29T17:55:05.960 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,957 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:05.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,957 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:05.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:05,958 [shard 0] osd - peering_event(id=607, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:06.069 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,068 [shard 0] alienstore - stat 2022-03-29T17:55:06.232 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:06.229+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:06.392 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:06,389 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:06.392 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:06,390 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:06.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:06.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=5118760856337372545 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:06.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:06.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=16580798734910078729 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:06.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:06.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:06.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:06.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:06,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:06.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:06.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:06,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:06.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,591 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:06.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:06.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:06.594 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:06,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:06.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:06.594 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:06,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:06.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:06.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:06.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:06.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:06,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:06.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:06,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:06.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:06,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:07.076 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:07,074 [shard 0] alienstore - stat 2022-03-29T17:55:07.794 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:07,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:07.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:07,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:07.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:07,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:07.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:07,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:07.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:07,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:07.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:07,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:08.082 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:08,081 [shard 0] alienstore - stat 2022-03-29T17:55:09.089 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:09,088 [shard 0] alienstore - stat 2022-03-29T17:55:09.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:09,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:09.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:09,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:09.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:09,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:09.397 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:09,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:09.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:09,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:09.397 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:09,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:09.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~181s 2022-03-29T17:55:09.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~154s 2022-03-29T17:55:10.096 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:10,095 [shard 0] alienstore - stat 2022-03-29T17:55:11.104 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:11,102 [shard 0] alienstore - stat 2022-03-29T17:55:11.233 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:11.230+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:11.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,565 [shard 0] osd - peering_event(id=608, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:11.567 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,565 [shard 0] osd - peering_event(id=608, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:11.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,565 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:55:11.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,565 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:55:11.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,565 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 334.198333740s) 2022-03-29T17:55:11.568 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,565 [shard 0] osd - log is not dirty 2022-03-29T17:55:11.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,565 [shard 0] osd - peering_event(id=608, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:11.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,565 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:11.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,566 [shard 0] osd - peering_event(id=608, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:11.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,575 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:11.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,575 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:11.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,575 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:55:11.577 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,575 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:55:11.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,575 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 334.208343506s) 2022-03-29T17:55:11.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,575 [shard 0] osd - log is not dirty 2022-03-29T17:55:11.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,575 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:11.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,575 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:11.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,576 [shard 0] osd - peering_event(id=609, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:11.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,605 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:11.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,605 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:11.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,605 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:55:11.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,605 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:55:11.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,605 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 334.238616943s) 2022-03-29T17:55:11.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,605 [shard 0] osd - log is not dirty 2022-03-29T17:55:11.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,605 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:11.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,605 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:11.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,606 [shard 0] osd - peering_event(id=610, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:11.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,635 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:11.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,635 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:11.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,635 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:55:11.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,635 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:55:11.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,636 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 334.268951416s) 2022-03-29T17:55:11.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,636 [shard 0] osd - log is not dirty 2022-03-29T17:55:11.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,636 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:11.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,636 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:11.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,636 [shard 0] osd - peering_event(id=611, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:11.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,645 [shard 0] osd - peering_event(id=612, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:11.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,645 [shard 0] osd - peering_event(id=612, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:11.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,645 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:55:11.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,645 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:55:11.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,645 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 334.278900146s) 2022-03-29T17:55:11.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,645 [shard 0] osd - log is not dirty 2022-03-29T17:55:11.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,645 [shard 0] osd - peering_event(id=612, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:11.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,645 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:11.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,646 [shard 0] osd - peering_event(id=612, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:11.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,675 [shard 0] osd - peering_event(id=613, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:11.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,675 [shard 0] osd - peering_event(id=613, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:11.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,676 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:55:11.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,676 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:55:11.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,676 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 334.309204102s) 2022-03-29T17:55:11.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,676 [shard 0] osd - log is not dirty 2022-03-29T17:55:11.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,676 [shard 0] osd - peering_event(id=613, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:11.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,676 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:11.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:11,676 [shard 0] osd - peering_event(id=613, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:12.110 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,109 [shard 0] alienstore - stat 2022-03-29T17:55:12.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=614, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:12.313 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=615, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:12.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=616, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:12.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=614, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:12.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=615, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:12.314 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=616, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:12.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:55:12.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 334.943542480s (sent 334.943542480s) 2022-03-29T17:55:12.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - log is not dirty 2022-03-29T17:55:12.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=615, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:12.315 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:12.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:55:12.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 334.943756104s (sent 334.943756104s) 2022-03-29T17:55:12.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - log is not dirty 2022-03-29T17:55:12.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=614, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:12.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:12.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:55:12.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 334.943908691s (sent 334.943908691s) 2022-03-29T17:55:12.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - log is not dirty 2022-03-29T17:55:12.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - peering_event(id=616, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:12.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,310 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:12.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,311 [shard 0] osd - peering_event(id=615, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:12.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,311 [shard 0] osd - peering_event(id=614, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:12.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,311 [shard 0] osd - peering_event(id=616, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:12.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:12.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:12.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:12.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:12.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:12.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:12.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:55:12.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 334.963745117s (sent 334.963745117s) 2022-03-29T17:55:12.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - log is not dirty 2022-03-29T17:55:12.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:12.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:12.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:55:12.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 334.963958740s (sent 334.963958740s) 2022-03-29T17:55:12.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - log is not dirty 2022-03-29T17:55:12.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:12.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:12.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,331 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:55:12.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,331 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 334.964141846s (sent 334.964141846s) 2022-03-29T17:55:12.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,331 [shard 0] osd - log is not dirty 2022-03-29T17:55:12.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,331 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:12.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:12.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,331 [shard 0] osd - peering_event(id=617, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:12.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,334 [shard 0] osd - peering_event(id=618, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:12.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:12,334 [shard 0] osd - peering_event(id=619, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:12.393 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:12,390 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:12.393 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:12,390 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:12.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:12.393 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,390 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=3504999921012071204 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:12.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:12.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,390 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=11765382292717164612 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:12.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:12.394 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:12.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:12.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:12,391 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:12.395 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:12.395 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:12,391 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:12.593 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,591 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:12.663 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:12.663 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,592 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:12.664 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:12,592 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:12.664 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:12.664 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:12,592 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:12.994 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,992 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:12.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,992 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:12.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,993 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:12.995 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:12,993 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:12.995 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:12,993 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:12.996 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:12,993 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:13.116 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:13,115 [shard 0] alienstore - stat 2022-03-29T17:55:13.795 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:13,793 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:13.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:13,793 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:13.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:13,794 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:13.796 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:13,794 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:13.796 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:13,794 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:13.797 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:13,794 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:13.954 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,950 [shard 0] osd - peering_event(id=620, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:13.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,950 [shard 0] osd - peering_event(id=620, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:13.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,951 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:55:13.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,951 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:55:13.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,951 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 336.584136963s) 2022-03-29T17:55:13.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,951 [shard 0] osd - log is not dirty 2022-03-29T17:55:13.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,951 [shard 0] osd - peering_event(id=620, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:13.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,951 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:13.958 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,952 [shard 0] osd - peering_event(id=620, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:13.964 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,960 [shard 0] osd - peering_event(id=621, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:13.964 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,961 [shard 0] osd - peering_event(id=621, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:13.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,961 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:55:13.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,961 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:55:13.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,961 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 336.594360352s) 2022-03-29T17:55:13.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,961 [shard 0] osd - log is not dirty 2022-03-29T17:55:13.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,961 [shard 0] osd - peering_event(id=621, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:13.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,961 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:13.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:13,962 [shard 0] osd - peering_event(id=621, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:14.123 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:14,122 [shard 0] alienstore - stat 2022-03-29T17:55:14.286 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:14,284 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:14.286 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:14,284 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:55:14.286 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:14,284 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:14.287 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:14,284 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:55:14.287 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:14,284 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:55:15.130 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:15,129 [shard 0] alienstore - stat 2022-03-29T17:55:15.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2022-03-29T17:55:15.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~159s 2022-03-29T17:55:15.396 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:15,394 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:15.397 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:15,394 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:15.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:15,395 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:15.399 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:15,395 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:15.399 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:15,395 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:15.399 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:15,395 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:16.138 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:16,136 [shard 0] alienstore - stat 2022-03-29T17:55:16.234 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:16.231+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:17.143 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:17,141 [shard 0] alienstore - stat 2022-03-29T17:55:17.871 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:17,869 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:55:17.872 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:17,870 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:55:17.872 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:17,870 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:17.872 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:17,870 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:55:17.873 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:17,870 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:55:18.149 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,148 [shard 0] alienstore - stat 2022-03-29T17:55:18.402 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:18,399 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:18.402 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:18,399 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:18.402 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,399 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:18.403 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,399 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=8407262350984858417 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:18.403 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,399 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:18.403 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,399 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=11457880961945598801 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:18.403 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,400 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:18.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,400 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:18.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,400 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:18.404 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:18,400 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:18.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,400 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:18.405 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:18,400 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:18.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,600 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:18.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,601 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:18.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,601 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:18.604 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:18,601 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:18.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:18,601 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:18.604 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:18,601 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:19.003 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,001 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:19.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,001 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:19.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,002 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:19.004 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:19,002 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:19.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,002 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:19.005 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:19,002 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:19.156 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,154 [shard 0] alienstore - stat 2022-03-29T17:55:19.569 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,567 [shard 0] osd - peering_event(id=622, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:19.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,567 [shard 0] osd - peering_event(id=622, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:19.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,567 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:55:19.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,567 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:55:19.570 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,567 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 342.200775146s) 2022-03-29T17:55:19.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,567 [shard 0] osd - log is not dirty 2022-03-29T17:55:19.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,567 [shard 0] osd - peering_event(id=622, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:19.571 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,567 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:19.572 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,568 [shard 0] osd - peering_event(id=622, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:19.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,577 [shard 0] osd - peering_event(id=623, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:19.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,577 [shard 0] osd - peering_event(id=623, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:19.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,577 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:55:19.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,577 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:55:19.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,577 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 342.210876465s) 2022-03-29T17:55:19.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,577 [shard 0] osd - log is not dirty 2022-03-29T17:55:19.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,577 [shard 0] osd - peering_event(id=623, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:19.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,577 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:19.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,578 [shard 0] osd - peering_event(id=623, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:19.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,607 [shard 0] osd - peering_event(id=624, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:19.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,607 [shard 0] osd - peering_event(id=624, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:19.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,607 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:55:19.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,607 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:55:19.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,607 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 342.240875244s) 2022-03-29T17:55:19.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,607 [shard 0] osd - log is not dirty 2022-03-29T17:55:19.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,607 [shard 0] osd - peering_event(id=624, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:19.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,607 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:19.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,608 [shard 0] osd - peering_event(id=624, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:19.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,637 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:19.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,637 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:19.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,638 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:55:19.641 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,638 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:55:19.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,638 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 342.271118164s) 2022-03-29T17:55:19.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,638 [shard 0] osd - log is not dirty 2022-03-29T17:55:19.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,638 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:19.642 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,638 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:19.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,638 [shard 0] osd - peering_event(id=625, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:19.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,647 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:19.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,648 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:19.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,648 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:55:19.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,648 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:55:19.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,648 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 342.281311035s) 2022-03-29T17:55:19.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,648 [shard 0] osd - log is not dirty 2022-03-29T17:55:19.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,648 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:19.652 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,648 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:19.652 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,649 [shard 0] osd - peering_event(id=626, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:19.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,678 [shard 0] osd - peering_event(id=627, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:19.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,678 [shard 0] osd - peering_event(id=627, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:19.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,678 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:55:19.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,678 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:55:19.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,678 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 342.311370850s) 2022-03-29T17:55:19.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,678 [shard 0] osd - log is not dirty 2022-03-29T17:55:19.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,678 [shard 0] osd - peering_event(id=627, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:19.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,678 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:19.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:19,679 [shard 0] osd - peering_event(id=627, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:19.805 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,802 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:19.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,802 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:19.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,803 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:19.806 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:19,803 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:19.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:19,803 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:19.807 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:19,803 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:20.163 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:20,161 [shard 0] alienstore - stat 2022-03-29T17:55:20.316 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - peering_event(id=628, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:20.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - peering_event(id=629, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:20.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - peering_event(id=630, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:20.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - peering_event(id=628, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:20.317 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - peering_event(id=630, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:20.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - peering_event(id=629, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:20.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:55:20.318 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 342.945800781s (sent 342.945800781s) 2022-03-29T17:55:20.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - log is not dirty 2022-03-29T17:55:20.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - peering_event(id=628, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:20.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:20.319 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,312 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:55:20.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 342.946014404s (sent 342.946014404s) 2022-03-29T17:55:20.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - log is not dirty 2022-03-29T17:55:20.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - peering_event(id=630, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:20.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:20.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:55:20.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 342.946228027s (sent 342.946228027s) 2022-03-29T17:55:20.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - log is not dirty 2022-03-29T17:55:20.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - peering_event(id=629, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:20.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:20.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - peering_event(id=628, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:20.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,313 [shard 0] osd - peering_event(id=630, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:20.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,316 [shard 0] osd - peering_event(id=629, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:20.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,332 [shard 0] osd - peering_event(id=631, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:20.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,332 [shard 0] osd - peering_event(id=632, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:20.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,332 [shard 0] osd - peering_event(id=633, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:20.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,332 [shard 0] osd - peering_event(id=631, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:20.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,332 [shard 0] osd - peering_event(id=633, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:20.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,332 [shard 0] osd - peering_event(id=632, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:20.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,332 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:55:20.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 342.966003418s (sent 342.966003418s) 2022-03-29T17:55:20.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - log is not dirty 2022-03-29T17:55:20.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - peering_event(id=631, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:20.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:20.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:55:20.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 342.966217041s (sent 342.966217041s) 2022-03-29T17:55:20.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - log is not dirty 2022-03-29T17:55:20.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - peering_event(id=633, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:20.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:20.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:55:20.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 342.966339111s (sent 342.966339111s) 2022-03-29T17:55:20.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - log is not dirty 2022-03-29T17:55:20.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - peering_event(id=632, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:20.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:20.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,333 [shard 0] osd - peering_event(id=631, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:20.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,336 [shard 0] osd - peering_event(id=632, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:20.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:20,336 [shard 0] osd - peering_event(id=633, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:20.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~191s 2022-03-29T17:55:20.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~164s 2022-03-29T17:55:21.170 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:21,168 [shard 0] alienstore - stat 2022-03-29T17:55:21.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:21.233+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:21.407 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:21,403 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:21.407 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:21,403 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:21.407 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:21,404 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:21.408 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:21,404 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:21.408 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:21,404 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:21.408 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:21,404 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:21.955 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,953 [shard 0] osd - peering_event(id=634, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:21.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,953 [shard 0] osd - peering_event(id=634, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:21.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,953 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:55:21.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,953 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:55:21.956 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,953 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 344.586669922s) 2022-03-29T17:55:21.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,953 [shard 0] osd - log is not dirty 2022-03-29T17:55:21.957 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,953 [shard 0] osd - peering_event(id=634, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:21.958 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,953 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:21.958 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,954 [shard 0] osd - peering_event(id=634, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:21.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,963 [shard 0] osd - peering_event(id=635, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:21.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,963 [shard 0] osd - peering_event(id=635, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:21.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,963 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:55:21.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,963 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:55:21.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,963 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 344.596710205s) 2022-03-29T17:55:21.967 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,963 [shard 0] osd - log is not dirty 2022-03-29T17:55:21.967 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,963 [shard 0] osd - peering_event(id=635, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:21.968 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,963 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:21.968 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:21,964 [shard 0] osd - peering_event(id=635, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:22.176 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:22,175 [shard 0] alienstore - stat 2022-03-29T17:55:23.183 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:23,182 [shard 0] alienstore - stat 2022-03-29T17:55:24.190 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,189 [shard 0] alienstore - stat 2022-03-29T17:55:24.403 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:24,400 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:24.403 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:24,400 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:24.403 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,400 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:24.403 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,400 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=9318028159510246518 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:24.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,400 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:24.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,400 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=3274832098060700052 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:24.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,401 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:24.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,401 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:24.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,401 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:24.405 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:24,401 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:24.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,401 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:24.405 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:24,401 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:24.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,601 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:24.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,601 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:24.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,602 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:24.605 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:24,602 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:24.605 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:24,602 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:24.605 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:24,602 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:25.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,002 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:25.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,003 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:25.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,003 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:25.006 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:25,003 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:25.006 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,003 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:25.006 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:25,003 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:25.198 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,196 [shard 0] alienstore - stat 2022-03-29T17:55:25.805 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,803 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:25.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,803 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:25.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,804 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:25.806 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:25,804 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:25.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:25,804 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:25.807 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:25,804 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:25.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~197s 2022-03-29T17:55:25.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~170s 2022-03-29T17:55:26.204 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:26,202 [shard 0] alienstore - stat 2022-03-29T17:55:26.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:26.234+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:27.211 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:27,209 [shard 0] alienstore - stat 2022-03-29T17:55:27.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:27,404 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:27.406 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:27,404 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:27.407 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:27,404 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:27.407 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:27,405 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:27.407 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:27,405 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:27.407 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:27,405 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:27.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,572 [shard 0] osd - peering_event(id=636, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:27.574 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,572 [shard 0] osd - peering_event(id=636, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:27.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,572 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:55:27.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,572 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:55:27.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,572 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 350.205718994s) 2022-03-29T17:55:27.575 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,572 [shard 0] osd - log is not dirty 2022-03-29T17:55:27.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,572 [shard 0] osd - peering_event(id=636, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:27.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,572 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:27.576 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,573 [shard 0] osd - peering_event(id=636, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:27.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,582 [shard 0] osd - peering_event(id=637, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:27.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,582 [shard 0] osd - peering_event(id=637, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:27.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,582 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:55:27.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,582 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:55:27.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,582 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 350.215576172s) 2022-03-29T17:55:27.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,582 [shard 0] osd - log is not dirty 2022-03-29T17:55:27.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,582 [shard 0] osd - peering_event(id=637, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:27.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,582 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:27.586 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,583 [shard 0] osd - peering_event(id=637, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:27.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,612 [shard 0] osd - peering_event(id=638, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:27.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,612 [shard 0] osd - peering_event(id=638, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:27.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,612 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:55:27.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,612 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:55:27.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,612 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 350.245819092s) 2022-03-29T17:55:27.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,612 [shard 0] osd - log is not dirty 2022-03-29T17:55:27.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,612 [shard 0] osd - peering_event(id=638, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:27.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,612 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:27.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,613 [shard 0] osd - peering_event(id=638, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:27.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,642 [shard 0] osd - peering_event(id=639, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:27.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,642 [shard 0] osd - peering_event(id=639, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:27.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,643 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:55:27.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,643 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:55:27.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,643 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 350.276092529s) 2022-03-29T17:55:27.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,643 [shard 0] osd - log is not dirty 2022-03-29T17:55:27.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,643 [shard 0] osd - peering_event(id=639, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:27.646 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,643 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:27.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,643 [shard 0] osd - peering_event(id=639, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:27.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,652 [shard 0] osd - peering_event(id=640, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:27.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,652 [shard 0] osd - peering_event(id=640, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:27.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,653 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:55:27.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,653 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:55:27.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,653 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 350.286132812s) 2022-03-29T17:55:27.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,653 [shard 0] osd - log is not dirty 2022-03-29T17:55:27.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,653 [shard 0] osd - peering_event(id=640, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:27.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,653 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:27.657 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,653 [shard 0] osd - peering_event(id=640, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:27.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,683 [shard 0] osd - peering_event(id=641, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:27.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,683 [shard 0] osd - peering_event(id=641, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:27.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,683 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:55:27.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,683 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:55:27.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,683 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 350.316406250s) 2022-03-29T17:55:27.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,683 [shard 0] osd - log is not dirty 2022-03-29T17:55:27.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,683 [shard 0] osd - peering_event(id=641, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:27.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,683 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:27.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:27,684 [shard 0] osd - peering_event(id=641, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:28.218 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:28,216 [shard 0] alienstore - stat 2022-03-29T17:55:28.320 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - peering_event(id=642, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:28.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - peering_event(id=643, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:28.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - peering_event(id=644, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:28.321 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - peering_event(id=642, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:28.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - peering_event(id=644, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:28.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - peering_event(id=643, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:28.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:55:28.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 350.950836182s (sent 350.950836182s) 2022-03-29T17:55:28.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - log is not dirty 2022-03-29T17:55:28.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - peering_event(id=642, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:28.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:28.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,317 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:55:28.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 350.951049805s (sent 350.951049805s) 2022-03-29T17:55:28.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - log is not dirty 2022-03-29T17:55:28.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - peering_event(id=644, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:28.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:28.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:55:28.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 350.951232910s (sent 350.951232910s) 2022-03-29T17:55:28.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - log is not dirty 2022-03-29T17:55:28.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - peering_event(id=643, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:28.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:28.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - peering_event(id=642, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:28.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,318 [shard 0] osd - peering_event(id=643, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:28.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,319 [shard 0] osd - peering_event(id=644, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:28.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - peering_event(id=645, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:28.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - peering_event(id=646, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:28.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - peering_event(id=647, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:28.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - peering_event(id=646, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:28.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - peering_event(id=645, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:28.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - peering_event(id=647, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:28.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:55:28.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 350.970916748s (sent 350.970916748s) 2022-03-29T17:55:28.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - log is not dirty 2022-03-29T17:55:28.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - peering_event(id=645, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:28.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,337 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:28.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:55:28.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 350.971069336s (sent 350.971069336s) 2022-03-29T17:55:28.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - log is not dirty 2022-03-29T17:55:28.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - peering_event(id=647, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:28.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:28.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:55:28.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 350.971221924s (sent 350.971221924s) 2022-03-29T17:55:28.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - log is not dirty 2022-03-29T17:55:28.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - peering_event(id=646, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:28.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:28.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,338 [shard 0] osd - peering_event(id=647, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:28.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,339 [shard 0] osd - peering_event(id=645, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:28.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:28,339 [shard 0] osd - peering_event(id=646, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:29.225 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:29,223 [shard 0] alienstore - stat 2022-03-29T17:55:29.286 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:29,284 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:29.286 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:29,285 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:55:29.287 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:29,285 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:29.287 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:29,285 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:55:29.287 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:29,285 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:55:29.960 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,958 [shard 0] osd - peering_event(id=648, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:29.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,958 [shard 0] osd - peering_event(id=648, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:29.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,958 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:55:29.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,958 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:55:29.961 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,958 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 352.591674805s) 2022-03-29T17:55:29.962 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,958 [shard 0] osd - log is not dirty 2022-03-29T17:55:29.962 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,958 [shard 0] osd - peering_event(id=648, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:29.962 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,958 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:29.962 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,959 [shard 0] osd - peering_event(id=648, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:29.970 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,968 [shard 0] osd - peering_event(id=649, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:29.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,968 [shard 0] osd - peering_event(id=649, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:29.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,968 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:55:29.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,968 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:55:29.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,968 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 352.601745605s) 2022-03-29T17:55:29.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,968 [shard 0] osd - log is not dirty 2022-03-29T17:55:29.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,968 [shard 0] osd - peering_event(id=649, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:29.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,968 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:29.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:29,969 [shard 0] osd - peering_event(id=649, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:30.231 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,230 [shard 0] alienstore - stat 2022-03-29T17:55:30.403 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:30,401 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:30.404 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:30,401 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:30.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,401 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:30.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,401 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=10732378713979106586 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:30.404 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,401 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:30.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,401 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=5476582080569138733 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:30.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,401 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:30.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,402 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:30.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,402 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:30.406 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:30,402 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:30.406 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,402 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:30.406 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:30,402 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:30.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,602 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:30.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,602 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:30.605 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,602 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:30.605 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:30,603 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:30.605 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:30,603 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:30.605 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:30,603 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:31.004 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,003 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:31.005 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,003 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:31.006 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,003 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:31.006 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:31,004 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:31.006 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,004 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:31.007 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:31,004 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:31.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~202s 2022-03-29T17:55:31.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~175s 2022-03-29T17:55:31.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:31.234+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:31.238 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,236 [shard 0] alienstore - stat 2022-03-29T17:55:31.805 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,804 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:31.806 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,804 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:31.807 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,804 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:31.807 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:31,805 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:31.807 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:31,805 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:31.807 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:31,805 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:32.245 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:32,243 [shard 0] alienstore - stat 2022-03-29T17:55:32.872 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:32,870 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:55:32.873 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:32,871 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:55:32.873 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:32,871 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:32.873 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:32,871 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:55:32.873 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:32,871 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:55:33.252 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:33,250 [shard 0] alienstore - stat 2022-03-29T17:55:33.406 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:33,405 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:33.407 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:33,405 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:33.408 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:33,405 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:33.408 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:33,405 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:33.408 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:33,405 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:33.408 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:33,405 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:34.259 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:34,257 [shard 0] alienstore - stat 2022-03-29T17:55:35.266 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:35,264 [shard 0] alienstore - stat 2022-03-29T17:55:35.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,576 [shard 0] osd - peering_event(id=650, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:35.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,576 [shard 0] osd - peering_event(id=650, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:35.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,576 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:55:35.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,576 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:55:35.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,576 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 358.209869385s) 2022-03-29T17:55:35.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,577 [shard 0] osd - log is not dirty 2022-03-29T17:55:35.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,577 [shard 0] osd - peering_event(id=650, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:35.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,577 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:35.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,577 [shard 0] osd - peering_event(id=650, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:35.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,586 [shard 0] osd - peering_event(id=651, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:35.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,586 [shard 0] osd - peering_event(id=651, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:35.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,586 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:55:35.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,586 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:55:35.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,586 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 358.219787598s) 2022-03-29T17:55:35.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,586 [shard 0] osd - log is not dirty 2022-03-29T17:55:35.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,586 [shard 0] osd - peering_event(id=651, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:35.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,586 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:35.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,587 [shard 0] osd - peering_event(id=651, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:35.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,616 [shard 0] osd - peering_event(id=652, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:35.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,616 [shard 0] osd - peering_event(id=652, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:35.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,616 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:55:35.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,616 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:55:35.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,616 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 358.249969482s) 2022-03-29T17:55:35.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,617 [shard 0] osd - log is not dirty 2022-03-29T17:55:35.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,617 [shard 0] osd - peering_event(id=652, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:35.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,617 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:35.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,617 [shard 0] osd - peering_event(id=652, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:35.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,646 [shard 0] osd - peering_event(id=653, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:35.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,647 [shard 0] osd - peering_event(id=653, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:35.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,647 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:55:35.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,647 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:55:35.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,647 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 358.280334473s) 2022-03-29T17:55:35.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,647 [shard 0] osd - log is not dirty 2022-03-29T17:55:35.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,647 [shard 0] osd - peering_event(id=653, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:35.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,647 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:35.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,648 [shard 0] osd - peering_event(id=653, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:35.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - peering_event(id=654, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:35.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - peering_event(id=654, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:35.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:55:35.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:55:35.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 358.290222168s) 2022-03-29T17:55:35.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - log is not dirty 2022-03-29T17:55:35.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - peering_event(id=654, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:35.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:35.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,657 [shard 0] osd - peering_event(id=654, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:35.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,687 [shard 0] osd - peering_event(id=655, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:35.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,687 [shard 0] osd - peering_event(id=655, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:35.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,687 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:55:35.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,687 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:55:35.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,687 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 358.320556641s) 2022-03-29T17:55:35.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,687 [shard 0] osd - log is not dirty 2022-03-29T17:55:35.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,687 [shard 0] osd - peering_event(id=655, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:35.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,687 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:35.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:35,688 [shard 0] osd - peering_event(id=655, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:36.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:36.235+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:36.270 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,268 [shard 0] alienstore - stat 2022-03-29T17:55:36.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,318 [shard 0] osd - peering_event(id=656, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:36.322 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,318 [shard 0] osd - peering_event(id=657, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:36.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,318 [shard 0] osd - peering_event(id=658, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:36.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - peering_event(id=656, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:36.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - peering_event(id=658, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:36.323 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - peering_event(id=657, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:36.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:55:36.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 358.952392578s (sent 358.952392578s) 2022-03-29T17:55:36.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - log is not dirty 2022-03-29T17:55:36.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - peering_event(id=658, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:36.324 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:36.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:55:36.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 358.952667236s (sent 358.952667236s) 2022-03-29T17:55:36.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - log is not dirty 2022-03-29T17:55:36.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - peering_event(id=657, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:36.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:36.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:55:36.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 358.952850342s (sent 358.952850342s) 2022-03-29T17:55:36.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - log is not dirty 2022-03-29T17:55:36.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - peering_event(id=656, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:36.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,319 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:36.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,320 [shard 0] osd - peering_event(id=658, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:36.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,320 [shard 0] osd - peering_event(id=657, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:36.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,320 [shard 0] osd - peering_event(id=656, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:36.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=659, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:36.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=660, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:36.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=661, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:36.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=659, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:36.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=660, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:36.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=661, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:36.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:55:36.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 358.972442627s (sent 358.972442627s) 2022-03-29T17:55:36.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - log is not dirty 2022-03-29T17:55:36.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=659, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:36.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:36.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:55:36.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 358.972656250s (sent 358.972656250s) 2022-03-29T17:55:36.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - log is not dirty 2022-03-29T17:55:36.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=661, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:36.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:36.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:55:36.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 358.972808838s (sent 358.972808838s) 2022-03-29T17:55:36.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - log is not dirty 2022-03-29T17:55:36.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - peering_event(id=660, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:36.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,339 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:36.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,340 [shard 0] osd - peering_event(id=661, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:36.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,340 [shard 0] osd - peering_event(id=659, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:36.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:36,340 [shard 0] osd - peering_event(id=660, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:36.411 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:36,409 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:36.411 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:36,409 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:36.412 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,409 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:36.412 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,409 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=9329900493357870661 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:36.412 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,409 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:36.412 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,409 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=4201255495599903522 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:36.412 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,410 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:36.413 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,410 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:36.414 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,410 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:36.414 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:36,410 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:36.414 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,410 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:36.414 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:36,410 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:36.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2022-03-29T17:55:36.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~180s 2022-03-29T17:55:36.612 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,610 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:36.613 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,611 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:36.613 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,611 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:36.613 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:36,611 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:36.614 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:36,611 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:36.614 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:36,611 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:37.013 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,011 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:37.013 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,011 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:37.014 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,012 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:37.015 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:37,012 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:37.015 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,012 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:37.015 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:37,012 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:37.276 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,275 [shard 0] alienstore - stat 2022-03-29T17:55:37.814 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,812 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:37.814 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,812 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:37.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,813 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:37.815 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:37,813 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:37.815 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:37,813 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:37.816 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:37,813 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:37.964 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,960 [shard 0] osd - peering_event(id=662, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:37.964 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,960 [shard 0] osd - peering_event(id=662, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:37.964 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,960 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:55:37.964 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,960 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:55:37.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,960 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 360.593444824s) 2022-03-29T17:55:37.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,960 [shard 0] osd - log is not dirty 2022-03-29T17:55:37.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,960 [shard 0] osd - peering_event(id=662, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:37.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,960 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:37.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,961 [shard 0] osd - peering_event(id=662, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:37.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,970 [shard 0] osd - peering_event(id=663, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:37.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,970 [shard 0] osd - peering_event(id=663, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:37.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,970 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:55:37.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,970 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:55:37.973 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,970 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 360.603363037s) 2022-03-29T17:55:37.973 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,970 [shard 0] osd - log is not dirty 2022-03-29T17:55:37.973 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,970 [shard 0] osd - peering_event(id=663, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:37.973 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,970 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:37.974 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:37,971 [shard 0] osd - peering_event(id=663, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:38.284 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:38,282 [shard 0] alienstore - stat 2022-03-29T17:55:39.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:39,288 [shard 0] alienstore - stat 2022-03-29T17:55:39.416 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:39,413 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:39.416 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:39,413 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:39.417 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:39,414 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:39.417 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:39,414 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:39.418 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:39,414 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:39.418 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:39,414 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:40.297 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:40,295 [shard 0] alienstore - stat 2022-03-29T17:55:41.239 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:41.236+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:41.303 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:41,302 [shard 0] alienstore - stat 2022-03-29T17:55:41.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~213s 2022-03-29T17:55:41.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~186s 2022-03-29T17:55:42.310 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,308 [shard 0] alienstore - stat 2022-03-29T17:55:42.421 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:42,419 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:42.422 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:42,419 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:42.422 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,419 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:42.422 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,419 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=6190538648834445794 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:42.423 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,419 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:42.423 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,419 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=2473518720377748885 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:42.423 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,420 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:42.424 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,420 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:42.424 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,420 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:42.424 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:42,420 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:42.424 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,420 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:42.424 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:42,420 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:42.622 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,620 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:42.622 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,620 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:42.623 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,621 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:42.623 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:42,621 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:42.624 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:42,621 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:42.624 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:42,621 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:43.023 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,021 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:43.023 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,021 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:43.024 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,022 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:43.024 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:43,022 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:43.024 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,022 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:43.024 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:43,022 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:43.316 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,315 [shard 0] alienstore - stat 2022-03-29T17:55:43.578 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,576 [shard 0] osd - peering_event(id=664, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:43.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,576 [shard 0] osd - peering_event(id=664, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:43.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,576 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:55:43.579 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,577 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:55:43.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,577 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 366.210021973s) 2022-03-29T17:55:43.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,577 [shard 0] osd - log is not dirty 2022-03-29T17:55:43.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,577 [shard 0] osd - peering_event(id=664, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:43.580 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,577 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:43.581 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,578 [shard 0] osd - peering_event(id=664, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:43.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,586 [shard 0] osd - peering_event(id=665, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:43.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,586 [shard 0] osd - peering_event(id=665, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:43.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,586 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:55:43.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,587 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:55:43.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,587 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 366.220062256s) 2022-03-29T17:55:43.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,587 [shard 0] osd - log is not dirty 2022-03-29T17:55:43.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,587 [shard 0] osd - peering_event(id=665, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:43.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,587 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:43.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,587 [shard 0] osd - peering_event(id=665, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:43.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,616 [shard 0] osd - peering_event(id=666, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:43.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,617 [shard 0] osd - peering_event(id=666, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:43.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,617 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:55:43.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,617 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:55:43.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,617 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 366.250183105s) 2022-03-29T17:55:43.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,617 [shard 0] osd - log is not dirty 2022-03-29T17:55:43.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,617 [shard 0] osd - peering_event(id=666, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:43.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,617 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:43.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,617 [shard 0] osd - peering_event(id=666, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:43.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,647 [shard 0] osd - peering_event(id=667, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:43.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,647 [shard 0] osd - peering_event(id=667, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:43.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,647 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:55:43.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,647 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:55:43.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,647 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 366.280395508s) 2022-03-29T17:55:43.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,647 [shard 0] osd - log is not dirty 2022-03-29T17:55:43.650 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,647 [shard 0] osd - peering_event(id=667, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:43.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,647 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:43.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,648 [shard 0] osd - peering_event(id=667, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:43.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,657 [shard 0] osd - peering_event(id=668, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:43.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,657 [shard 0] osd - peering_event(id=668, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:43.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,657 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:55:43.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,657 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:55:43.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,657 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 366.290435791s) 2022-03-29T17:55:43.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,657 [shard 0] osd - log is not dirty 2022-03-29T17:55:43.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,657 [shard 0] osd - peering_event(id=668, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:43.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,657 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:43.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,658 [shard 0] osd - peering_event(id=668, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:43.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,687 [shard 0] osd - peering_event(id=669, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:43.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,687 [shard 0] osd - peering_event(id=669, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:43.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,687 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:55:43.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,687 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:55:43.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,687 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 366.320617676s) 2022-03-29T17:55:43.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,687 [shard 0] osd - log is not dirty 2022-03-29T17:55:43.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,687 [shard 0] osd - peering_event(id=669, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:43.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,687 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:43.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:43,688 [shard 0] osd - peering_event(id=669, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:43.824 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,822 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:43.824 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,822 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:43.825 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,823 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:43.825 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:43,823 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:43.825 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:43,823 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:43.826 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:43,823 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:44.287 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:44,285 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:44.288 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:44,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:55:44.288 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:44,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:44.288 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:44,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:55:44.288 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:44,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:55:44.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,321 [shard 0] osd - peering_event(id=670, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:44.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,321 [shard 0] osd - peering_event(id=671, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:44.325 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,321 [shard 0] osd - peering_event(id=672, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:44.326 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:44,322 [shard 0] alienstore - stat 2022-03-29T17:55:44.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - peering_event(id=670, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:44.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - peering_event(id=672, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:44.326 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - peering_event(id=671, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:44.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:55:44.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 366.955505371s (sent 366.955505371s) 2022-03-29T17:55:44.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - log is not dirty 2022-03-29T17:55:44.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - peering_event(id=672, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:44.327 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:44.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:55:44.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 366.955688477s (sent 366.955688477s) 2022-03-29T17:55:44.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - log is not dirty 2022-03-29T17:55:44.328 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - peering_event(id=670, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:44.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:44.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:55:44.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 366.955932617s (sent 366.955932617s) 2022-03-29T17:55:44.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - log is not dirty 2022-03-29T17:55:44.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - peering_event(id=671, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:44.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,322 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:44.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,323 [shard 0] osd - peering_event(id=670, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:44.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,323 [shard 0] osd - peering_event(id=672, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:44.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,323 [shard 0] osd - peering_event(id=671, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:44.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,341 [shard 0] osd - peering_event(id=673, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:44.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,341 [shard 0] osd - peering_event(id=674, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:44.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,341 [shard 0] osd - peering_event(id=675, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:44.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,341 [shard 0] osd - peering_event(id=673, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:44.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - peering_event(id=674, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:44.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - peering_event(id=675, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:44.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:55:44.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 366.975341797s (sent 366.975341797s) 2022-03-29T17:55:44.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - log is not dirty 2022-03-29T17:55:44.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - peering_event(id=674, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:44.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:44.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:55:44.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 366.975494385s (sent 366.975494385s) 2022-03-29T17:55:44.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - log is not dirty 2022-03-29T17:55:44.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - peering_event(id=675, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:44.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:44.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:55:44.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 366.975708008s (sent 366.975708008s) 2022-03-29T17:55:44.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - log is not dirty 2022-03-29T17:55:44.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - peering_event(id=673, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:44.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,342 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:44.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,343 [shard 0] osd - peering_event(id=674, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:44.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,343 [shard 0] osd - peering_event(id=675, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:44.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:44,343 [shard 0] osd - peering_event(id=673, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:45.330 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:45,328 [shard 0] alienstore - stat 2022-03-29T17:55:45.424 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:45,423 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:45.425 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:45,423 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:45.425 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:45,423 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:45.426 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:45,423 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:45.426 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:45,424 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:45.426 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:45,424 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:45.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,962 [shard 0] osd - peering_event(id=676, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:45.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,962 [shard 0] osd - peering_event(id=676, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:45.965 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,962 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:55:45.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,963 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:55:45.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,963 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 368.596038818s) 2022-03-29T17:55:45.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,963 [shard 0] osd - log is not dirty 2022-03-29T17:55:45.966 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,963 [shard 0] osd - peering_event(id=676, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:45.967 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,963 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:45.967 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,964 [shard 0] osd - peering_event(id=676, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:45.974 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,972 [shard 0] osd - peering_event(id=677, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:45.975 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,972 [shard 0] osd - peering_event(id=677, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:45.975 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,973 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:55:45.975 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,973 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:55:45.975 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,973 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 368.606079102s) 2022-03-29T17:55:45.976 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,973 [shard 0] osd - log is not dirty 2022-03-29T17:55:45.976 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,973 [shard 0] osd - peering_event(id=677, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:45.976 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,973 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:45.976 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:45,973 [shard 0] osd - peering_event(id=677, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:46.240 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:46.237+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:46.336 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:46,335 [shard 0] alienstore - stat 2022-03-29T17:55:47.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~218s 2022-03-29T17:55:47.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~191s 2022-03-29T17:55:47.343 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:47,342 [shard 0] alienstore - stat 2022-03-29T17:55:47.873 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:47,871 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:55:47.874 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:47,871 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:55:47.874 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:47,872 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:47.874 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:47,872 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:55:47.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:47,872 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:55:48.350 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,348 [shard 0] alienstore - stat 2022-03-29T17:55:48.431 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:48,429 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:48.432 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:48,429 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:48.432 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,429 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:48.432 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,429 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=16987904181853527472 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:48.433 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,429 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:48.433 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,429 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=13214069010738578965 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:48.433 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,429 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:48.433 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,430 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:48.433 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,430 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:48.434 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:48,430 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:48.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,430 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:48.434 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:48,430 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:48.632 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,630 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:48.633 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,630 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:48.633 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,630 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:48.633 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:48,631 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:48.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:48,631 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:48.634 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:48,631 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:49.033 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,031 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:49.033 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,031 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:49.034 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,031 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:49.034 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:49,031 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:49.034 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,031 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:49.034 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:49,031 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:49.357 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,355 [shard 0] alienstore - stat 2022-03-29T17:55:49.834 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,832 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:49.834 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,832 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:49.834 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,832 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:49.835 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:49,832 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:49.835 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:49,832 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:49.835 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:49,832 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:50.364 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:50,362 [shard 0] alienstore - stat 2022-03-29T17:55:51.241 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:51.238+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:51.371 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:51,369 [shard 0] alienstore - stat 2022-03-29T17:55:51.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:51,432 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:51.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:51,433 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:51.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:51,433 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:51.436 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:51,433 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:51.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:51,433 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:51.436 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:51,433 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:51.583 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,580 [shard 0] osd - peering_event(id=678, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:51.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,581 [shard 0] osd - peering_event(id=678, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:51.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,581 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:55:51.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,581 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:55:51.584 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,581 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 374.214324951s) 2022-03-29T17:55:51.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,581 [shard 0] osd - log is not dirty 2022-03-29T17:55:51.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,581 [shard 0] osd - peering_event(id=678, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:51.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,581 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:51.585 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,582 [shard 0] osd - peering_event(id=678, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:51.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,590 [shard 0] osd - peering_event(id=679, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:51.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,591 [shard 0] osd - peering_event(id=679, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:51.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,591 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:55:51.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,591 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:55:51.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,591 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 374.224182129s) 2022-03-29T17:55:51.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,591 [shard 0] osd - log is not dirty 2022-03-29T17:55:51.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,591 [shard 0] osd - peering_event(id=679, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:51.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,591 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:51.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,591 [shard 0] osd - peering_event(id=679, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:51.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,621 [shard 0] osd - peering_event(id=680, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:51.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,621 [shard 0] osd - peering_event(id=680, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:51.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,621 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:55:51.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,621 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:55:51.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,621 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 374.254455566s) 2022-03-29T17:55:51.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,621 [shard 0] osd - log is not dirty 2022-03-29T17:55:51.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,621 [shard 0] osd - peering_event(id=680, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:51.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,621 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:51.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,622 [shard 0] osd - peering_event(id=680, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:51.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,651 [shard 0] osd - peering_event(id=681, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:51.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,651 [shard 0] osd - peering_event(id=681, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:51.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,651 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:55:51.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,651 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:55:51.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,651 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 374.284881592s) 2022-03-29T17:55:51.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,651 [shard 0] osd - log is not dirty 2022-03-29T17:55:51.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,651 [shard 0] osd - peering_event(id=681, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:51.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,651 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:51.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,652 [shard 0] osd - peering_event(id=681, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:51.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,661 [shard 0] osd - peering_event(id=682, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:51.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,661 [shard 0] osd - peering_event(id=682, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:51.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,661 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:55:51.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,661 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:55:51.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,661 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 374.294952393s) 2022-03-29T17:55:51.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,662 [shard 0] osd - log is not dirty 2022-03-29T17:55:51.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,662 [shard 0] osd - peering_event(id=682, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:51.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,662 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:51.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,662 [shard 0] osd - peering_event(id=682, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:51.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,691 [shard 0] osd - peering_event(id=683, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:51.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,691 [shard 0] osd - peering_event(id=683, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:51.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,691 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:55:51.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,692 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:55:51.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,692 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 374.325012207s) 2022-03-29T17:55:51.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,692 [shard 0] osd - log is not dirty 2022-03-29T17:55:51.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,692 [shard 0] osd - peering_event(id=683, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:51.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,692 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:51.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:51,692 [shard 0] osd - peering_event(id=683, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:52.329 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,325 [shard 0] osd - peering_event(id=684, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:52.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - peering_event(id=685, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:52.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - peering_event(id=686, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:52.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - peering_event(id=684, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:52.330 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - peering_event(id=685, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:52.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - peering_event(id=686, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:52.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:55:52.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 374.959442139s (sent 374.959442139s) 2022-03-29T17:55:52.331 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - log is not dirty 2022-03-29T17:55:52.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - peering_event(id=685, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:52.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:52.332 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:55:52.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 374.959716797s (sent 374.959716797s) 2022-03-29T17:55:52.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - log is not dirty 2022-03-29T17:55:52.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - peering_event(id=686, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:52.333 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:52.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:55:52.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 374.959869385s (sent 374.959869385s) 2022-03-29T17:55:52.334 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - log is not dirty 2022-03-29T17:55:52.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - peering_event(id=684, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:52.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,326 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:52.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,327 [shard 0] osd - peering_event(id=685, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:52.335 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,327 [shard 0] osd - peering_event(id=686, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:52.336 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,327 [shard 0] osd - peering_event(id=684, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:52.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - peering_event(id=687, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:52.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - peering_event(id=688, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:52.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - peering_event(id=689, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:55:52.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - peering_event(id=687, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:52.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - peering_event(id=689, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:52.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - peering_event(id=688, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:55:52.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:55:52.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 374.979614258s (sent 374.979614258s) 2022-03-29T17:55:52.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - log is not dirty 2022-03-29T17:55:52.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - peering_event(id=687, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:52.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:52.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:55:52.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 374.979858398s (sent 374.979858398s) 2022-03-29T17:55:52.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - log is not dirty 2022-03-29T17:55:52.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - peering_event(id=689, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:52.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:52.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:55:52.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 374.979980469s (sent 374.979980469s) 2022-03-29T17:55:52.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,346 [shard 0] osd - log is not dirty 2022-03-29T17:55:52.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,347 [shard 0] osd - peering_event(id=688, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:55:52.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,347 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:52.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,347 [shard 0] osd - peering_event(id=687, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:52.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,347 [shard 0] osd - peering_event(id=688, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:52.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:52,347 [shard 0] osd - peering_event(id=689, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:55:52.377 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:52,376 [shard 0] alienstore - stat 2022-03-29T17:55:52.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~223s 2022-03-29T17:55:52.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~196s 2022-03-29T17:55:53.385 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:53,383 [shard 0] alienstore - stat 2022-03-29T17:55:53.970 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,967 [shard 0] osd - peering_event(id=690, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:53.970 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,967 [shard 0] osd - peering_event(id=690, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:53.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,967 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:55:53.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,967 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:55:53.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,967 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 376.600463867s) 2022-03-29T17:55:53.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,967 [shard 0] osd - log is not dirty 2022-03-29T17:55:53.971 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,967 [shard 0] osd - peering_event(id=690, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:53.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,967 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:53.972 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,968 [shard 0] osd - peering_event(id=690, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:53.979 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,977 [shard 0] osd - peering_event(id=691, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:55:53.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,977 [shard 0] osd - peering_event(id=691, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:55:53.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,977 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:55:53.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,977 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:55:53.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,977 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 376.610473633s) 2022-03-29T17:55:53.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,977 [shard 0] osd - log is not dirty 2022-03-29T17:55:53.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,977 [shard 0] osd - peering_event(id=691, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:55:53.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,977 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:53.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:53,978 [shard 0] osd - peering_event(id=691, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:55:54.150 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,148 [shard 0] monc - sending auth(proto 2 165 bytes epoch 0) v1 2022-03-29T17:55:54.150 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,148 [shard 0] monc - waiting 2022-03-29T17:55:54.151 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,149 [shard 0] monc - handle_auth_reply mon v2:172.21.15.138:6803/35579 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:55:54.151 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,149 [shard 0] monc - handle_auth_reply 2022-03-29T17:55:54.151 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,149 [shard 0] monc - do_auth_single: mon v2:172.21.15.138:6803/35579 => v2:172.21.15.138:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 2022-03-29T17:55:54.392 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,390 [shard 0] alienstore - stat 2022-03-29T17:55:54.433 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:55:54,430 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:55:54.434 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:54,430 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:55:54.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,430 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:55:54.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,430 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=11685293672813130526 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:54.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,430 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:55:54.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,430 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=11456295423706039911 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:55:54.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,430 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:54.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:55:54.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:54.436 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:54,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:55:54.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:54.437 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:54,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:55:54.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,631 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:54.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,632 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:54.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,632 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:54.636 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:54,632 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:55:54.636 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:54,632 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:54.637 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:54,632 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:55:55.034 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,032 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:55.035 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,032 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:55.035 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,033 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:55.035 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:55,033 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:55:55.036 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,033 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:55.036 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:55,033 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:55:55.398 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,397 [shard 0] alienstore - stat 2022-03-29T17:55:55.835 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,833 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:55.835 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,833 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:55.836 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:55.836 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:55,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:55:55.837 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:55,834 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:55.837 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:55,834 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:55:56.242 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:55:56.239+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:55:56.405 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:56,404 [shard 0] alienstore - stat 2022-03-29T17:55:56.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,761 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4127.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): start 2022-03-29T17:55:56.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,761 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4127.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): in repeat 2022-03-29T17:55:56.766 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4127.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]) same_interval_since: 11 2022-03-29T17:55:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - OpSequencer::start_op: op=13, last_started=11, last_unblocked=11, last_completed=11 2022-03-29T17:55:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - do_recover_missing check for recovery, 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:55:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4127.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): got obc lock 2022-03-29T17:55:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - with_head_obc 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:55:56.767 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - with_head_obc: found 2:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:55:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:55:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - handling op omap-get-vals on object 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:55:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:55:56.768 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,762 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8 - object 2:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:55:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,763 [shard 0] osd - with_head_obc: released 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:55:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,763 [shard 0] osd - OpSequencer::finish_op_in_order: op=13, last_started=13, last_unblocked=13, last_completed=11 2022-03-29T17:55:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,763 [shard 0] osd - client_request(id=13, detail=m=[osd_op(client.4127.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): destroying 2022-03-29T17:55:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,763 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4127.0:13 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): start 2022-03-29T17:55:56.769 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,763 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4127.0:13 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): in repeat 2022-03-29T17:55:56.770 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,763 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4127.0:13 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]) same_interval_since: 14 2022-03-29T17:55:56.770 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,763 [shard 0] osd - OpSequencer::start_op: op=14, last_started=12, last_unblocked=12, last_completed=12 2022-03-29T17:55:56.770 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,763 [shard 0] osd - do_recover_missing check for recovery, 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:55:56.770 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4127.0:13 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): got obc lock 2022-03-29T17:55:56.771 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - with_head_obc 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:55:56.771 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - with_head_obc: found 3:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:55:56.771 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:55:56.771 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - handling op omap-get-vals on object 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:55:56.771 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:55:56.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:13 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8 - object 3:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:55:56.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - with_head_obc: released 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:55:56.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - OpSequencer::finish_op_in_order: op=14, last_started=14, last_unblocked=14, last_completed=12 2022-03-29T17:55:56.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:56,764 [shard 0] osd - client_request(id=14, detail=m=[osd_op(client.4127.0:13 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): destroying 2022-03-29T17:55:57.412 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:57,410 [shard 0] alienstore - stat 2022-03-29T17:55:57.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:57,434 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:57.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:57,434 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:57.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:57,435 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:57.437 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:57,435 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:55:57.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:57,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:57.437 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:57,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:55:57.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~229s 2022-03-29T17:55:57.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~202s 2022-03-29T17:55:58.419 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:58,417 [shard 0] alienstore - stat 2022-03-29T17:55:59.288 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:59,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:55:59.288 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:59,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:55:59.288 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:59,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:55:59.289 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:55:59,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:55:59.289 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:59,286 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:55:59.426 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:55:59,424 [shard 0] alienstore - stat 2022-03-29T17:55:59.587 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,585 [shard 0] osd - peering_event(id=692, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:59.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,585 [shard 0] osd - peering_event(id=692, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:59.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,585 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:55:59.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,585 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:55:59.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,585 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 382.218688965s) 2022-03-29T17:55:59.588 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,585 [shard 0] osd - log is not dirty 2022-03-29T17:55:59.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,585 [shard 0] osd - peering_event(id=692, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:59.589 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,585 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:59.590 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,586 [shard 0] osd - peering_event(id=692, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:59.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,595 [shard 0] osd - peering_event(id=693, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:59.597 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,595 [shard 0] osd - peering_event(id=693, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:59.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,595 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:55:59.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,595 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:55:59.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,595 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 382.228637695s) 2022-03-29T17:55:59.598 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,595 [shard 0] osd - log is not dirty 2022-03-29T17:55:59.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,595 [shard 0] osd - peering_event(id=693, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:59.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,595 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:59.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,596 [shard 0] osd - peering_event(id=693, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:59.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,625 [shard 0] osd - peering_event(id=694, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:59.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,625 [shard 0] osd - peering_event(id=694, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:59.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,625 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:55:59.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,626 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:55:59.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,626 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 382.259033203s) 2022-03-29T17:55:59.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,626 [shard 0] osd - log is not dirty 2022-03-29T17:55:59.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,626 [shard 0] osd - peering_event(id=694, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:59.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,626 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:59.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,626 [shard 0] osd - peering_event(id=694, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:59.658 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,655 [shard 0] osd - peering_event(id=695, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:59.658 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,655 [shard 0] osd - peering_event(id=695, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:59.658 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,656 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:55:59.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,656 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:55:59.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,656 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 382.289245605s) 2022-03-29T17:55:59.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,656 [shard 0] osd - log is not dirty 2022-03-29T17:55:59.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,656 [shard 0] osd - peering_event(id=695, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:59.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,656 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:59.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,656 [shard 0] osd - peering_event(id=695, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:59.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,665 [shard 0] osd - peering_event(id=696, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:59.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,666 [shard 0] osd - peering_event(id=696, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:59.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,666 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:55:59.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,666 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:55:59.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,666 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 382.299346924s) 2022-03-29T17:55:59.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,666 [shard 0] osd - log is not dirty 2022-03-29T17:55:59.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,666 [shard 0] osd - peering_event(id=696, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:59.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,666 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:59.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,667 [shard 0] osd - peering_event(id=696, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:55:59.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,696 [shard 0] osd - peering_event(id=697, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:55:59.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,696 [shard 0] osd - peering_event(id=697, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:55:59.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,696 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:55:59.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,696 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:55:59.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,696 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 382.329467773s) 2022-03-29T17:55:59.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,696 [shard 0] osd - log is not dirty 2022-03-29T17:55:59.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,696 [shard 0] osd - peering_event(id=697, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:55:59.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,696 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:55:59.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:55:59,697 [shard 0] osd - peering_event(id=697, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:00.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - peering_event(id=698, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:00.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - peering_event(id=699, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:00.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - peering_event(id=700, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:00.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - peering_event(id=700, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:00.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - peering_event(id=698, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:00.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - peering_event(id=699, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:00.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:56:00.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 382.963867188s (sent 382.963867188s) 2022-03-29T17:56:00.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - log is not dirty 2022-03-29T17:56:00.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - peering_event(id=698, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:00.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,330 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:00.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:56:00.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 382.964080811s (sent 382.964080811s) 2022-03-29T17:56:00.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - log is not dirty 2022-03-29T17:56:00.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - peering_event(id=700, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:00.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:00.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:56:00.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 382.964233398s (sent 382.964233398s) 2022-03-29T17:56:00.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - log is not dirty 2022-03-29T17:56:00.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - peering_event(id=699, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:00.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:00.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - peering_event(id=698, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:00.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - peering_event(id=699, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:00.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,331 [shard 0] osd - peering_event(id=700, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:00.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - peering_event(id=701, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:00.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - peering_event(id=702, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:00.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - peering_event(id=703, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:00.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - peering_event(id=701, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:00.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - peering_event(id=702, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:00.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - peering_event(id=703, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:00.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:56:00.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 382.983825684s (sent 382.983825684s) 2022-03-29T17:56:00.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - log is not dirty 2022-03-29T17:56:00.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - peering_event(id=701, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:00.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:00.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:56:00.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,350 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 382.984008789s (sent 382.984008789s) 2022-03-29T17:56:00.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - log is not dirty 2022-03-29T17:56:00.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - peering_event(id=703, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:56:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 382.984161377s (sent 382.984161377s) 2022-03-29T17:56:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - log is not dirty 2022-03-29T17:56:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - peering_event(id=702, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,351 [shard 0] osd - peering_event(id=703, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,352 [shard 0] osd - peering_event(id=701, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:00.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:00,352 [shard 0] osd - peering_event(id=702, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:00.433 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:00,430 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:00.434 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:00,430 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:00.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:00.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=2198026180215176890 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:00.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:00.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=8706076965113485128 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:00.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,431 [shard 0] alienstore - stat 2022-03-29T17:56:00.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:00.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:00.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:00.436 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:00,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:00.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,432 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:00.437 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:00,432 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:00.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,632 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,632 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,633 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:00,633 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:00.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:00,633 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:00.636 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:00,633 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:01.035 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,033 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:01.035 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,033 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:01.036 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:01.036 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:01,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:01.036 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,034 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:01.036 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:01,034 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:01.243 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:01.240+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:01.439 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,438 [shard 0] alienstore - stat 2022-03-29T17:56:01.835 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:01.836 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,834 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:01.837 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:01.837 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:01,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:01.837 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:01,835 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:01.837 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:01,835 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:01.974 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,971 [shard 0] osd - peering_event(id=704, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:01.974 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,971 [shard 0] osd - peering_event(id=704, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:01.974 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,971 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:56:01.975 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,971 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:56:01.975 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,971 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 384.604827881s) 2022-03-29T17:56:01.975 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,971 [shard 0] osd - log is not dirty 2022-03-29T17:56:01.975 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,971 [shard 0] osd - peering_event(id=704, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:01.976 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,971 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:01.976 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,972 [shard 0] osd - peering_event(id=704, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:01.983 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,981 [shard 0] osd - peering_event(id=705, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:01.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,981 [shard 0] osd - peering_event(id=705, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:01.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,981 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:56:01.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,982 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:56:01.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,982 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 384.615020752s) 2022-03-29T17:56:01.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,982 [shard 0] osd - log is not dirty 2022-03-29T17:56:01.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,982 [shard 0] osd - peering_event(id=705, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:01.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,982 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:01.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:01,982 [shard 0] osd - peering_event(id=705, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:02.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:02,445 [shard 0] alienstore - stat 2022-03-29T17:56:02.874 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:02,872 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:56:02.874 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:02,872 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:56:02.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:02,872 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:02.875 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:02,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:56:02.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:02,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:56:02.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~234s 2022-03-29T17:56:02.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~207s 2022-03-29T17:56:03.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:03,435 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:03.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:03,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:03.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:03,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:03.438 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:03,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:03.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:03,435 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:03.438 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:03,436 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:03.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:03,451 [shard 0] alienstore - stat 2022-03-29T17:56:04.460 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:04,458 [shard 0] alienstore - stat 2022-03-29T17:56:05.466 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:05,465 [shard 0] alienstore - stat 2022-03-29T17:56:06.244 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:06.242+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:06.434 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:06,431 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:06.434 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:06,431 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:06.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:06.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=6860670417285032986 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:06.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:06.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=12055195061691361291 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:06.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,432 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:06.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:06.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:06.436 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:06,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:06.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,432 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:06.437 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:06,432 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:06.473 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,471 [shard 0] alienstore - stat 2022-03-29T17:56:06.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,632 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:06.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,633 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:06.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,633 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:06.636 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:06,633 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:06.636 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:06,633 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:06.636 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:06,633 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:07.035 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,033 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:07.035 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:07.036 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,034 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:07.036 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:07,034 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:07.037 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:07.037 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:07,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:07.479 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,478 [shard 0] alienstore - stat 2022-03-29T17:56:07.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,588 [shard 0] osd - peering_event(id=706, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:07.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,588 [shard 0] osd - peering_event(id=706, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:07.591 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,589 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:56:07.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,589 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:56:07.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,589 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 390.222198486s) 2022-03-29T17:56:07.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,589 [shard 0] osd - log is not dirty 2022-03-29T17:56:07.592 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,589 [shard 0] osd - peering_event(id=706, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:07.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,589 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:07.593 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,590 [shard 0] osd - peering_event(id=706, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:07.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,599 [shard 0] osd - peering_event(id=707, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:07.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,599 [shard 0] osd - peering_event(id=707, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:07.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,599 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:56:07.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,599 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:56:07.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,599 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 390.232421875s) 2022-03-29T17:56:07.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,599 [shard 0] osd - log is not dirty 2022-03-29T17:56:07.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,599 [shard 0] osd - peering_event(id=707, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:07.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,599 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:07.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,600 [shard 0] osd - peering_event(id=707, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:07.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,629 [shard 0] osd - peering_event(id=708, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:07.631 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,629 [shard 0] osd - peering_event(id=708, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:07.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,629 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:56:07.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,629 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:56:07.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,629 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 390.262512207s) 2022-03-29T17:56:07.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,629 [shard 0] osd - log is not dirty 2022-03-29T17:56:07.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,629 [shard 0] osd - peering_event(id=708, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:07.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,629 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:07.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,630 [shard 0] osd - peering_event(id=708, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:07.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,659 [shard 0] osd - peering_event(id=709, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:07.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,659 [shard 0] osd - peering_event(id=709, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:07.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,659 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:56:07.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,659 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:56:07.662 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,659 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 390.292633057s) 2022-03-29T17:56:07.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,659 [shard 0] osd - log is not dirty 2022-03-29T17:56:07.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,659 [shard 0] osd - peering_event(id=709, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:07.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,659 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:07.663 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,660 [shard 0] osd - peering_event(id=709, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:07.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,669 [shard 0] osd - peering_event(id=710, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:07.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,669 [shard 0] osd - peering_event(id=710, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:07.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,669 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:56:07.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,669 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:56:07.672 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,669 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 390.302703857s) 2022-03-29T17:56:07.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,669 [shard 0] osd - log is not dirty 2022-03-29T17:56:07.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,669 [shard 0] osd - peering_event(id=710, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:07.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,669 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:07.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,670 [shard 0] osd - peering_event(id=710, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:07.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,699 [shard 0] osd - peering_event(id=711, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:07.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,699 [shard 0] osd - peering_event(id=711, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:07.702 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,699 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:56:07.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,699 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:56:07.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,700 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 390.332977295s) 2022-03-29T17:56:07.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,700 [shard 0] osd - log is not dirty 2022-03-29T17:56:07.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,700 [shard 0] osd - peering_event(id=711, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:07.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,700 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:07.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:07,700 [shard 0] osd - peering_event(id=711, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:07.836 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,834 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:07.836 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:07.837 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,835 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:07.837 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:07,835 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:07.838 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:07,835 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:07.838 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:07,835 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:08.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~239s 2022-03-29T17:56:08.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~212s 2022-03-29T17:56:08.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,333 [shard 0] osd - peering_event(id=712, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:08.337 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,333 [shard 0] osd - peering_event(id=713, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:08.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,333 [shard 0] osd - peering_event(id=714, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:08.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,333 [shard 0] osd - peering_event(id=713, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:08.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,333 [shard 0] osd - peering_event(id=714, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:08.338 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - peering_event(id=712, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:08.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:56:08.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 390.967224121s (sent 390.967224121s) 2022-03-29T17:56:08.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - log is not dirty 2022-03-29T17:56:08.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - peering_event(id=712, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:08.339 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:08.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:56:08.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 390.967437744s (sent 390.967437744s) 2022-03-29T17:56:08.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - log is not dirty 2022-03-29T17:56:08.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - peering_event(id=714, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:08.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:08.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:56:08.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 390.967620850s (sent 390.967620850s) 2022-03-29T17:56:08.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - log is not dirty 2022-03-29T17:56:08.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - peering_event(id=713, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:08.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,334 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:08.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,335 [shard 0] osd - peering_event(id=712, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:08.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,335 [shard 0] osd - peering_event(id=714, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:08.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,335 [shard 0] osd - peering_event(id=713, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:08.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,353 [shard 0] osd - peering_event(id=715, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:08.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - peering_event(id=716, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:08.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - peering_event(id=717, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:08.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - peering_event(id=716, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:08.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - peering_event(id=717, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:08.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - peering_event(id=715, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:08.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:56:08.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 390.987487793s (sent 390.987487793s) 2022-03-29T17:56:08.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - log is not dirty 2022-03-29T17:56:08.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - peering_event(id=715, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:08.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:08.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:56:08.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 390.987670898s (sent 390.987670898s) 2022-03-29T17:56:08.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - log is not dirty 2022-03-29T17:56:08.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - peering_event(id=716, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:08.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:08.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:56:08.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 390.987854004s (sent 390.987854004s) 2022-03-29T17:56:08.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - log is not dirty 2022-03-29T17:56:08.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - peering_event(id=717, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:08.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,354 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:08.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,355 [shard 0] osd - peering_event(id=715, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:08.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,355 [shard 0] osd - peering_event(id=716, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:08.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:08,355 [shard 0] osd - peering_event(id=717, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:08.486 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:08,484 [shard 0] alienstore - stat 2022-03-29T17:56:09.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:09,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:09.440 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:09,435 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:09.441 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:09,436 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:09.441 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:09,436 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:09.441 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:09,436 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:09.442 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:09,436 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:09.493 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:09,491 [shard 0] alienstore - stat 2022-03-29T17:56:09.977 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,974 [shard 0] osd - peering_event(id=718, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:09.977 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,974 [shard 0] osd - peering_event(id=718, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:09.977 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,975 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:56:09.978 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,975 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:56:09.978 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,975 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 392.608123779s) 2022-03-29T17:56:09.978 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,975 [shard 0] osd - log is not dirty 2022-03-29T17:56:09.978 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,975 [shard 0] osd - peering_event(id=718, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:09.978 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,975 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:09.979 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,976 [shard 0] osd - peering_event(id=718, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:09.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,984 [shard 0] osd - peering_event(id=719, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:09.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,984 [shard 0] osd - peering_event(id=719, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:09.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,985 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:56:09.987 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,985 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:56:09.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,985 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 392.618194580s) 2022-03-29T17:56:09.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,985 [shard 0] osd - log is not dirty 2022-03-29T17:56:09.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,985 [shard 0] osd - peering_event(id=719, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:09.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,985 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:09.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:09,985 [shard 0] osd - peering_event(id=719, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:10.500 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:10,498 [shard 0] alienstore - stat 2022-03-29T17:56:11.245 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:11.243+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:11.506 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:11,505 [shard 0] alienstore - stat 2022-03-29T17:56:12.433 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:12,431 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:12.434 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:12,431 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:12.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:12.434 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,431 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=68562244084420950 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:12.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:12.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=9939870809714145419 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:12.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,431 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:12.435 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:12.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:12.436 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:12,432 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:12.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,432 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:12.436 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:12,432 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:12.513 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,511 [shard 0] alienstore - stat 2022-03-29T17:56:12.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,632 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:12.634 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,632 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:12.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,633 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:12.635 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:12,633 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:12.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:12,633 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:12.636 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:12,633 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:13.036 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,033 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:13.036 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,033 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:13.037 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,034 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:13.037 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:13,034 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:13.038 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:13.038 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:13,034 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:13.520 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,518 [shard 0] alienstore - stat 2022-03-29T17:56:13.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~244s 2022-03-29T17:56:13.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~217s 2022-03-29T17:56:13.835 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,834 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:13.836 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:13.837 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:13.837 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:13,834 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:13.837 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:13,835 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:13.837 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:13,835 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:14.289 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:14,287 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:14.289 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:14,287 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:56:14.289 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:14,287 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:14.290 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:14,287 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:56:14.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:14,287 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:56:14.526 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:14,525 [shard 0] alienstore - stat 2022-03-29T17:56:15.436 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:15,435 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:15.437 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:15,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:15.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:15,435 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:15.438 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:15,435 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:15.438 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:15,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:15.438 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:15,435 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:15.533 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:15,531 [shard 0] alienstore - stat 2022-03-29T17:56:15.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,592 [shard 0] osd - peering_event(id=720, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:15.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,592 [shard 0] osd - peering_event(id=720, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:15.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,592 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:56:15.594 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,592 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:56:15.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,592 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 398.225372314s) 2022-03-29T17:56:15.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,592 [shard 0] osd - log is not dirty 2022-03-29T17:56:15.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,592 [shard 0] osd - peering_event(id=720, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:15.595 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,592 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:15.596 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,593 [shard 0] osd - peering_event(id=720, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:15.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,602 [shard 0] osd - peering_event(id=721, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:15.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,602 [shard 0] osd - peering_event(id=721, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:15.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,602 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:56:15.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,602 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:56:15.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,602 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 398.235504150s) 2022-03-29T17:56:15.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,602 [shard 0] osd - log is not dirty 2022-03-29T17:56:15.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,602 [shard 0] osd - peering_event(id=721, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:15.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,602 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:15.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,603 [shard 0] osd - peering_event(id=721, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:15.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,632 [shard 0] osd - peering_event(id=722, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:15.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,632 [shard 0] osd - peering_event(id=722, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:15.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,632 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:56:15.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,632 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:56:15.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,632 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 398.265747070s) 2022-03-29T17:56:15.636 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,632 [shard 0] osd - log is not dirty 2022-03-29T17:56:15.636 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,632 [shard 0] osd - peering_event(id=722, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:15.636 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,632 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:15.636 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,633 [shard 0] osd - peering_event(id=722, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:15.664 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,662 [shard 0] osd - peering_event(id=723, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:15.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,662 [shard 0] osd - peering_event(id=723, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:15.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,662 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:56:15.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,662 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:56:15.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,662 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 398.295898438s) 2022-03-29T17:56:15.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,662 [shard 0] osd - log is not dirty 2022-03-29T17:56:15.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,663 [shard 0] osd - peering_event(id=723, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:15.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,663 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:15.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,663 [shard 0] osd - peering_event(id=723, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:15.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,672 [shard 0] osd - peering_event(id=724, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:15.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,672 [shard 0] osd - peering_event(id=724, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:15.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,672 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:56:15.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,672 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:56:15.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,672 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 398.305908203s) 2022-03-29T17:56:15.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,672 [shard 0] osd - log is not dirty 2022-03-29T17:56:15.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,672 [shard 0] osd - peering_event(id=724, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:15.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,673 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:15.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,673 [shard 0] osd - peering_event(id=724, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:15.706 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,702 [shard 0] osd - peering_event(id=725, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:15.706 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,702 [shard 0] osd - peering_event(id=725, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:15.706 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,703 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:56:15.706 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,703 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:56:15.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,703 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 398.336151123s) 2022-03-29T17:56:15.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,703 [shard 0] osd - log is not dirty 2022-03-29T17:56:15.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,703 [shard 0] osd - peering_event(id=725, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:15.707 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,703 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:15.708 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:15,703 [shard 0] osd - peering_event(id=725, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:16.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:16.244+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:16.340 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,336 [shard 0] osd - peering_event(id=726, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:16.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,336 [shard 0] osd - peering_event(id=727, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:16.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - peering_event(id=728, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:16.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - peering_event(id=727, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:16.341 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - peering_event(id=726, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:16.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - peering_event(id=728, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:16.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:56:16.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 398.970489502s (sent 398.970489502s) 2022-03-29T17:56:16.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - log is not dirty 2022-03-29T17:56:16.342 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - peering_event(id=727, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:16.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:16.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:56:16.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 398.970794678s (sent 398.970794678s) 2022-03-29T17:56:16.343 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - log is not dirty 2022-03-29T17:56:16.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - peering_event(id=728, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:16.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:16.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:56:16.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 398.970916748s (sent 398.970916748s) 2022-03-29T17:56:16.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - log is not dirty 2022-03-29T17:56:16.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - peering_event(id=726, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:16.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,337 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:16.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,338 [shard 0] osd - peering_event(id=727, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:16.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,338 [shard 0] osd - peering_event(id=728, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:16.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,338 [shard 0] osd - peering_event(id=726, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:16.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,356 [shard 0] osd - peering_event(id=729, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:16.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - peering_event(id=730, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:16.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - peering_event(id=731, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:16.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - peering_event(id=730, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:16.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - peering_event(id=731, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:16.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - peering_event(id=729, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:16.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:56:16.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 398.990539551s (sent 398.990539551s) 2022-03-29T17:56:16.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - log is not dirty 2022-03-29T17:56:16.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - peering_event(id=731, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:16.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:16.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:56:16.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 398.990722656s (sent 398.990722656s) 2022-03-29T17:56:16.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - log is not dirty 2022-03-29T17:56:16.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - peering_event(id=730, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:16.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:16.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:56:16.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 398.990875244s (sent 398.990875244s) 2022-03-29T17:56:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - log is not dirty 2022-03-29T17:56:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - peering_event(id=729, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,357 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,358 [shard 0] osd - peering_event(id=731, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:16.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,358 [shard 0] osd - peering_event(id=729, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:16.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:16,358 [shard 0] osd - peering_event(id=730, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:16.539 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:16,538 [shard 0] alienstore - stat 2022-03-29T17:56:17.546 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:17,545 [shard 0] alienstore - stat 2022-03-29T17:56:17.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:17,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:56:17.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:17,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:56:17.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:17,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:17.876 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:17,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:56:17.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:17,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:56:17.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,978 [shard 0] osd - peering_event(id=732, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:17.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,978 [shard 0] osd - peering_event(id=732, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:17.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,978 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:56:17.980 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,978 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:56:17.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,978 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 400.611450195s) 2022-03-29T17:56:17.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,978 [shard 0] osd - log is not dirty 2022-03-29T17:56:17.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,978 [shard 0] osd - peering_event(id=732, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:17.981 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,978 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:17.982 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,979 [shard 0] osd - peering_event(id=732, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:17.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,988 [shard 0] osd - peering_event(id=733, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:17.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,988 [shard 0] osd - peering_event(id=733, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:17.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,988 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:56:17.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,988 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:56:17.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,988 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 400.621337891s) 2022-03-29T17:56:17.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,988 [shard 0] osd - log is not dirty 2022-03-29T17:56:17.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,988 [shard 0] osd - peering_event(id=733, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:17.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,988 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:17.992 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:17,989 [shard 0] osd - peering_event(id=733, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:18.444 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:18,441 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:18.445 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:18,441 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:18.445 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,441 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:18.445 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,441 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=6925881972247431335 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:18.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,441 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:18.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,441 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=7415904365911697720 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:18.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,441 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:18.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,441 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:18.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,442 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:18.447 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:18,442 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:18.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,442 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:18.448 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:18,442 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:18.553 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,551 [shard 0] alienstore - stat 2022-03-29T17:56:18.644 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,642 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:18.644 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,642 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:18.645 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,642 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:18.645 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:18,642 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:18.645 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:18,643 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:18.646 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:18,643 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:18.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~250s 2022-03-29T17:56:18.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~223s 2022-03-29T17:56:19.045 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,043 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:19.045 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,043 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:19.046 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,043 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:19.046 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:19,043 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:19.046 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,043 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:19.046 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:19,043 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:19.560 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,558 [shard 0] alienstore - stat 2022-03-29T17:56:19.845 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,843 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:19.845 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,844 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:19.846 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,844 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:19.847 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:19,844 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:19.847 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:19,844 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:19.847 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:19,844 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:20.567 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:20,565 [shard 0] alienstore - stat 2022-03-29T17:56:21.247 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:21.244+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:21.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:21,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:21.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:21,445 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:21.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:21,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:21.448 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:21,446 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:21.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:21,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:21.448 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:21,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:21.573 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:21,572 [shard 0] alienstore - stat 2022-03-29T17:56:22.580 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:22,579 [shard 0] alienstore - stat 2022-03-29T17:56:23.587 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:23,586 [shard 0] alienstore - stat 2022-03-29T17:56:23.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,595 [shard 0] osd - peering_event(id=734, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:23.599 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,596 [shard 0] osd - peering_event(id=734, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:23.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,596 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:56:23.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,596 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:56:23.600 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,596 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 406.229217529s) 2022-03-29T17:56:23.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,596 [shard 0] osd - log is not dirty 2022-03-29T17:56:23.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,596 [shard 0] osd - peering_event(id=734, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:23.601 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,596 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:23.602 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,597 [shard 0] osd - peering_event(id=734, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:23.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,606 [shard 0] osd - peering_event(id=735, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:23.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,606 [shard 0] osd - peering_event(id=735, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:23.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,606 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:56:23.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,606 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:56:23.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,606 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 406.239410400s) 2022-03-29T17:56:23.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,606 [shard 0] osd - log is not dirty 2022-03-29T17:56:23.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,606 [shard 0] osd - peering_event(id=735, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:23.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,606 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:23.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,607 [shard 0] osd - peering_event(id=735, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:23.638 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,636 [shard 0] osd - peering_event(id=736, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:23.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,636 [shard 0] osd - peering_event(id=736, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:23.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,636 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:56:23.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,636 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:56:23.639 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,636 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 406.269439697s) 2022-03-29T17:56:23.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,636 [shard 0] osd - log is not dirty 2022-03-29T17:56:23.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,636 [shard 0] osd - peering_event(id=736, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:23.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,636 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:23.640 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,637 [shard 0] osd - peering_event(id=736, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:23.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,666 [shard 0] osd - peering_event(id=737, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:23.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,666 [shard 0] osd - peering_event(id=737, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:23.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,666 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:56:23.669 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,666 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:56:23.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,666 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 406.299835205s) 2022-03-29T17:56:23.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,666 [shard 0] osd - log is not dirty 2022-03-29T17:56:23.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,666 [shard 0] osd - peering_event(id=737, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:23.670 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,666 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:23.671 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,667 [shard 0] osd - peering_event(id=737, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:23.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,676 [shard 0] osd - peering_event(id=738, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:23.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,676 [shard 0] osd - peering_event(id=738, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,676 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:56:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,676 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:56:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,676 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 406.309661865s) 2022-03-29T17:56:23.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,676 [shard 0] osd - log is not dirty 2022-03-29T17:56:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,676 [shard 0] osd - peering_event(id=738, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,676 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,677 [shard 0] osd - peering_event(id=738, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:23.708 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,706 [shard 0] osd - peering_event(id=739, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:23.709 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,706 [shard 0] osd - peering_event(id=739, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:23.709 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,706 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:56:23.709 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,706 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:56:23.709 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,706 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 406.339935303s) 2022-03-29T17:56:23.709 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,707 [shard 0] osd - log is not dirty 2022-03-29T17:56:23.710 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,707 [shard 0] osd - peering_event(id=739, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:23.710 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,707 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:23.710 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:23,707 [shard 0] osd - peering_event(id=739, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:24.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2022-03-29T17:56:24.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~228s 2022-03-29T17:56:24.344 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - peering_event(id=740, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:24.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - peering_event(id=741, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:24.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - peering_event(id=742, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:24.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - peering_event(id=741, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:24.345 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - peering_event(id=742, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:24.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - peering_event(id=740, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:24.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:56:24.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 406.974731445s (sent 406.974731445s) 2022-03-29T17:56:24.346 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - log is not dirty 2022-03-29T17:56:24.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - peering_event(id=741, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:24.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:24.347 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:56:24.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 406.974945068s (sent 406.974945068s) 2022-03-29T17:56:24.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - log is not dirty 2022-03-29T17:56:24.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - peering_event(id=742, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:24.348 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,341 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:24.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,342 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:56:24.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,342 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 406.975189209s (sent 406.975189209s) 2022-03-29T17:56:24.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,342 [shard 0] osd - log is not dirty 2022-03-29T17:56:24.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,342 [shard 0] osd - peering_event(id=740, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:24.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,342 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:24.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,342 [shard 0] osd - peering_event(id=741, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:24.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,342 [shard 0] osd - peering_event(id=742, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:24.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,343 [shard 0] osd - peering_event(id=740, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:24.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - peering_event(id=743, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:24.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - peering_event(id=744, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:24.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - peering_event(id=745, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:24.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - peering_event(id=744, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:24.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - peering_event(id=743, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:24.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - peering_event(id=745, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:24.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:56:24.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 406.994750977s (sent 406.994750977s) 2022-03-29T17:56:24.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - log is not dirty 2022-03-29T17:56:24.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - peering_event(id=744, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:24.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:24.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:56:24.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 406.994934082s (sent 406.994934082s) 2022-03-29T17:56:24.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,361 [shard 0] osd - log is not dirty 2022-03-29T17:56:24.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - peering_event(id=743, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:24.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:24.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:56:24.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 406.995239258s (sent 406.995239258s) 2022-03-29T17:56:24.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - log is not dirty 2022-03-29T17:56:24.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - peering_event(id=745, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:24.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:24.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - peering_event(id=744, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:24.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,362 [shard 0] osd - peering_event(id=743, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:24.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:24,363 [shard 0] osd - peering_event(id=745, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:24.444 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:24,441 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:24.445 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:24,441 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:24.445 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,441 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:24.445 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,441 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=5859895445154854209 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:24.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,442 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:24.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,442 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=3440399712183665601 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:24.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,442 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:24.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,442 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:24.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,442 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:24.447 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:24,442 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:24.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:24.447 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:24,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:24.594 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,593 [shard 0] alienstore - stat 2022-03-29T17:56:24.645 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,643 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:24.645 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,643 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:24.645 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,643 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:24.646 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:24,643 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:24.646 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:24,643 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:24.646 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:24,643 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:25.046 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,043 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:25.046 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,044 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:25.046 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,044 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:25.047 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:25,044 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:25.047 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,044 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:25.047 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:25,044 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:25.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,599 [shard 0] alienstore - stat 2022-03-29T17:56:25.847 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,844 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:25.847 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,845 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:25.848 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,845 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:25.848 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:25,845 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:25.848 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:25,845 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:25.848 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:25,845 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:25.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,982 [shard 0] osd - peering_event(id=746, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:25.984 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,982 [shard 0] osd - peering_event(id=746, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:25.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,982 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:56:25.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,982 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:56:25.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,982 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 408.615447998s) 2022-03-29T17:56:25.985 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,982 [shard 0] osd - log is not dirty 2022-03-29T17:56:25.986 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,982 [shard 0] osd - peering_event(id=746, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:25.986 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,982 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:25.986 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,983 [shard 0] osd - peering_event(id=746, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:25.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,992 [shard 0] osd - peering_event(id=747, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:25.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,992 [shard 0] osd - peering_event(id=747, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:25.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,992 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:56:25.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,992 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:56:25.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,992 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 408.625366211s) 2022-03-29T17:56:25.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,992 [shard 0] osd - log is not dirty 2022-03-29T17:56:25.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,992 [shard 0] osd - peering_event(id=747, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:25.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,992 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:25.996 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:25,993 [shard 0] osd - peering_event(id=747, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:26.248 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:26.245+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:26.607 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:26,606 [shard 0] alienstore - stat 2022-03-29T17:56:27.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:27,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:27.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:27,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:27.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:27,446 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:27.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:27,446 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:27.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:27,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:27.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:27,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:27.615 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:27,613 [shard 0] alienstore - stat 2022-03-29T17:56:28.621 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:28,620 [shard 0] alienstore - stat 2022-03-29T17:56:29.289 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:29,287 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:29.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:29,288 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:56:29.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:29,288 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:29.290 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:29,288 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:56:29.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:29,288 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:56:29.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2022-03-29T17:56:29.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~233s 2022-03-29T17:56:29.629 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:29,627 [shard 0] alienstore - stat 2022-03-29T17:56:30.445 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:30,442 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:30.445 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:30,442 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:30.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,442 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:30.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,442 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=7436295550586834053 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:30.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,442 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:30.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,443 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=6772530618464672397 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:30.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:30.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,443 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:30.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,443 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:30.447 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:30,443 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:30.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:30.448 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:30,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:30.635 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,634 [shard 0] alienstore - stat 2022-03-29T17:56:30.646 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,644 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:30.646 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,644 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:30.646 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,644 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:30.646 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:30,644 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:30.647 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:30,644 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:30.647 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:30,644 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:31.047 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,044 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:31.047 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,045 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:31.047 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,045 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:31.048 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:31,045 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:31.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,045 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:31.048 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:31,045 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:31.248 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:31.246+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:31.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,600 [shard 0] osd - peering_event(id=748, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:31.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,600 [shard 0] osd - peering_event(id=748, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:31.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,600 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:56:31.603 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,601 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:56:31.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,601 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 414.234039307s) 2022-03-29T17:56:31.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,601 [shard 0] osd - log is not dirty 2022-03-29T17:56:31.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,601 [shard 0] osd - peering_event(id=748, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:31.604 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,601 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:31.605 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,602 [shard 0] osd - peering_event(id=748, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:31.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,610 [shard 0] osd - peering_event(id=749, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:31.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,610 [shard 0] osd - peering_event(id=749, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:31.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,611 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:56:31.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,611 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:56:31.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,611 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 414.244110107s) 2022-03-29T17:56:31.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,611 [shard 0] osd - log is not dirty 2022-03-29T17:56:31.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,611 [shard 0] osd - peering_event(id=749, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:31.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,611 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:31.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,611 [shard 0] osd - peering_event(id=749, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:31.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,640 [shard 0] osd - peering_event(id=750, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:31.643 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,641 [shard 0] alienstore - stat 2022-03-29T17:56:31.643 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,641 [shard 0] osd - peering_event(id=750, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:31.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,641 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:56:31.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,641 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:56:31.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,641 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 414.274597168s) 2022-03-29T17:56:31.644 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,641 [shard 0] osd - log is not dirty 2022-03-29T17:56:31.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,641 [shard 0] osd - peering_event(id=750, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:31.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,641 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:31.645 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,642 [shard 0] osd - peering_event(id=750, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:31.673 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,671 [shard 0] osd - peering_event(id=751, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:31.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,671 [shard 0] osd - peering_event(id=751, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:31.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,671 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:56:31.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,671 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:56:31.674 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,671 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 414.304504395s) 2022-03-29T17:56:31.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,671 [shard 0] osd - log is not dirty 2022-03-29T17:56:31.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,671 [shard 0] osd - peering_event(id=751, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:31.675 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,671 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:31.676 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,672 [shard 0] osd - peering_event(id=751, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:31.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,681 [shard 0] osd - peering_event(id=752, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:31.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,681 [shard 0] osd - peering_event(id=752, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:31.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,681 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:56:31.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,681 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:56:31.684 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,681 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 414.314514160s) 2022-03-29T17:56:31.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,681 [shard 0] osd - log is not dirty 2022-03-29T17:56:31.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,681 [shard 0] osd - peering_event(id=752, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:31.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,681 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:31.685 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,682 [shard 0] osd - peering_event(id=752, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:31.713 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,711 [shard 0] osd - peering_event(id=753, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:31.714 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,711 [shard 0] osd - peering_event(id=753, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:31.714 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,711 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:56:31.714 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,711 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:56:31.714 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,711 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 414.344696045s) 2022-03-29T17:56:31.715 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,711 [shard 0] osd - log is not dirty 2022-03-29T17:56:31.715 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,711 [shard 0] osd - peering_event(id=753, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:31.715 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,711 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:31.716 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:31,712 [shard 0] osd - peering_event(id=753, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:31.847 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,845 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:31.847 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,845 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:31.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,846 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:31.849 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:31,846 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:31.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:31,846 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:31.850 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:31,846 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:32.349 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,345 [shard 0] osd - peering_event(id=754, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:32.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,345 [shard 0] osd - peering_event(id=755, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:32.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - peering_event(id=756, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:32.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - peering_event(id=756, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:32.350 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - peering_event(id=754, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:32.351 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - peering_event(id=755, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:32.351 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:56:32.351 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 414.979461670s (sent 414.979461670s) 2022-03-29T17:56:32.351 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - log is not dirty 2022-03-29T17:56:32.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - peering_event(id=754, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:32.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:32.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:56:32.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 414.979705811s (sent 414.979705811s) 2022-03-29T17:56:32.352 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - log is not dirty 2022-03-29T17:56:32.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - peering_event(id=755, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:32.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:32.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:56:32.353 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 414.979949951s (sent 414.979949951s) 2022-03-29T17:56:32.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - log is not dirty 2022-03-29T17:56:32.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - peering_event(id=756, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:32.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,346 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:32.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,347 [shard 0] osd - peering_event(id=754, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:32.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,347 [shard 0] osd - peering_event(id=755, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:32.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,347 [shard 0] osd - peering_event(id=756, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:32.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,365 [shard 0] osd - peering_event(id=757, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:32.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,365 [shard 0] osd - peering_event(id=758, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:32.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - peering_event(id=759, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:32.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - peering_event(id=758, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:32.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - peering_event(id=759, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:32.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - peering_event(id=757, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:32.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:56:32.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 414.999420166s (sent 414.999420166s) 2022-03-29T17:56:32.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - log is not dirty 2022-03-29T17:56:32.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - peering_event(id=759, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:32.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:32.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:56:32.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 414.999633789s (sent 414.999633789s) 2022-03-29T17:56:32.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - log is not dirty 2022-03-29T17:56:32.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - peering_event(id=758, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:32.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:32.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:56:32.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 414.999816895s (sent 414.999816895s) 2022-03-29T17:56:32.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - log is not dirty 2022-03-29T17:56:32.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - peering_event(id=757, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:32.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,366 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:32.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,367 [shard 0] osd - peering_event(id=759, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:32.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,367 [shard 0] osd - peering_event(id=757, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:32.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:32,367 [shard 0] osd - peering_event(id=758, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:32.649 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:32,647 [shard 0] alienstore - stat 2022-03-29T17:56:32.875 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:32,873 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:56:32.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:32,874 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:56:32.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:32,874 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:32.876 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:32,874 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:56:32.877 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:32,874 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:56:33.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:33,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:33.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:33,446 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:33.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:33,447 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:33.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:33,447 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:33.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:33,447 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:33.450 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:33,447 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:33.656 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:33,654 [shard 0] alienstore - stat 2022-03-29T17:56:33.988 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,986 [shard 0] osd - peering_event(id=760, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:33.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,986 [shard 0] osd - peering_event(id=760, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:33.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,986 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:56:33.989 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,987 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:56:33.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,987 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 416.620056152s) 2022-03-29T17:56:33.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,987 [shard 0] osd - log is not dirty 2022-03-29T17:56:33.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,987 [shard 0] osd - peering_event(id=760, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:33.990 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,987 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:33.991 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,988 [shard 0] osd - peering_event(id=760, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:33.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,996 [shard 0] osd - peering_event(id=761, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:33.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,996 [shard 0] osd - peering_event(id=761, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:33.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,996 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:56:33.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,996 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:56:33.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,996 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 416.629943848s) 2022-03-29T17:56:33.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,997 [shard 0] osd - log is not dirty 2022-03-29T17:56:34.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,997 [shard 0] osd - peering_event(id=761, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:34.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,997 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:34.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:33,997 [shard 0] osd - peering_event(id=761, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:34.663 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:34,661 [shard 0] alienstore - stat 2022-03-29T17:56:34.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~266s 2022-03-29T17:56:34.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~239s 2022-03-29T17:56:35.669 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:35,668 [shard 0] alienstore - stat 2022-03-29T17:56:36.250 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:36.247+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:36.446 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:36,443 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:36.446 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:36,443 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:36.446 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:36.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=17240552537768385812 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:36.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,443 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:36.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,443 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=12803960370311459782 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:36.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:36.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:36.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:36.448 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:36,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:36.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:36.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:36,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:36.646 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,644 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:36.646 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,644 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:36.647 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:36.647 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:36,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:36.648 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:36.648 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:36,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:36.676 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:36,675 [shard 0] alienstore - stat 2022-03-29T17:56:37.047 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,045 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:37.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,045 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:37.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:37.048 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:37,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:37.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:37.049 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:37,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:37.683 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,681 [shard 0] alienstore - stat 2022-03-29T17:56:37.848 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,846 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:37.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,846 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:37.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,846 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:37.849 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:37,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:37.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:37,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:37.850 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:37,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:38.689 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:38,688 [shard 0] alienstore - stat 2022-03-29T17:56:39.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:39,447 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:39.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:39,447 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:39.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:39,447 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:39.450 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:39,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:39.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:39,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:39.451 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:39,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:39.606 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,604 [shard 0] osd - peering_event(id=762, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:39.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,604 [shard 0] osd - peering_event(id=762, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:39.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,604 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:56:39.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,604 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:56:39.607 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,604 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 422.237884521s) 2022-03-29T17:56:39.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,605 [shard 0] osd - log is not dirty 2022-03-29T17:56:39.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,605 [shard 0] osd - peering_event(id=762, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:39.608 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,605 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:39.609 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,605 [shard 0] osd - peering_event(id=762, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:39.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,614 [shard 0] osd - peering_event(id=763, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:39.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,614 [shard 0] osd - peering_event(id=763, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:39.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,614 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:56:39.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,614 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:56:39.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,614 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 422.247802734s) 2022-03-29T17:56:39.617 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,614 [shard 0] osd - log is not dirty 2022-03-29T17:56:39.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,614 [shard 0] osd - peering_event(id=763, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:39.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,614 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:39.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,615 [shard 0] osd - peering_event(id=763, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:39.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,644 [shard 0] osd - peering_event(id=764, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:39.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,644 [shard 0] osd - peering_event(id=764, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:39.647 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,645 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:56:39.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,645 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:56:39.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,645 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 422.278076172s) 2022-03-29T17:56:39.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,645 [shard 0] osd - log is not dirty 2022-03-29T17:56:39.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,645 [shard 0] osd - peering_event(id=764, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:39.648 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,645 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:39.649 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,645 [shard 0] osd - peering_event(id=764, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:39.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,674 [shard 0] osd - peering_event(id=765, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:39.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,675 [shard 0] osd - peering_event(id=765, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:39.677 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,675 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:56:39.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,675 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:56:39.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,675 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 422.308227539s) 2022-03-29T17:56:39.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,675 [shard 0] osd - log is not dirty 2022-03-29T17:56:39.678 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,675 [shard 0] osd - peering_event(id=765, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:39.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,675 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:39.679 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,675 [shard 0] osd - peering_event(id=765, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:39.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,685 [shard 0] osd - peering_event(id=766, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:39.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,685 [shard 0] osd - peering_event(id=766, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:39.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,685 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:56:39.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,685 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:56:39.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,685 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 422.318511963s) 2022-03-29T17:56:39.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,685 [shard 0] osd - log is not dirty 2022-03-29T17:56:39.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,685 [shard 0] osd - peering_event(id=766, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:39.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,685 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:39.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,686 [shard 0] osd - peering_event(id=766, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:39.696 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:39,695 [shard 0] alienstore - stat 2022-03-29T17:56:39.717 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,715 [shard 0] osd - peering_event(id=767, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:39.718 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,715 [shard 0] osd - peering_event(id=767, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:39.718 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,715 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:56:39.718 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,715 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:56:39.719 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,715 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 422.348571777s) 2022-03-29T17:56:39.719 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,715 [shard 0] osd - log is not dirty 2022-03-29T17:56:39.719 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,715 [shard 0] osd - peering_event(id=767, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:39.720 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,715 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:39.720 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:39,716 [shard 0] osd - peering_event(id=767, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:40.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~271s 2022-03-29T17:56:40.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~244s 2022-03-29T17:56:40.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,349 [shard 0] osd - peering_event(id=768, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:40.354 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,349 [shard 0] osd - peering_event(id=769, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:40.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,349 [shard 0] osd - peering_event(id=770, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:40.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,349 [shard 0] osd - peering_event(id=769, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:40.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,349 [shard 0] osd - peering_event(id=770, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:40.355 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,349 [shard 0] osd - peering_event(id=768, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:40.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:56:40.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 422.983154297s (sent 422.983154297s) 2022-03-29T17:56:40.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - log is not dirty 2022-03-29T17:56:40.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - peering_event(id=769, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:40.356 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:40.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:56:40.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 422.983398438s (sent 422.983398438s) 2022-03-29T17:56:40.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - log is not dirty 2022-03-29T17:56:40.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - peering_event(id=768, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:40.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:40.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:56:40.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 422.983642578s (sent 422.983642578s) 2022-03-29T17:56:40.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - log is not dirty 2022-03-29T17:56:40.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - peering_event(id=770, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:40.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,350 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:40.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,351 [shard 0] osd - peering_event(id=769, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:40.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,351 [shard 0] osd - peering_event(id=768, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:40.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,351 [shard 0] osd - peering_event(id=770, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:40.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,369 [shard 0] osd - peering_event(id=771, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:40.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,369 [shard 0] osd - peering_event(id=772, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:40.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,369 [shard 0] osd - peering_event(id=773, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:40.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,369 [shard 0] osd - peering_event(id=771, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:40.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,369 [shard 0] osd - peering_event(id=773, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:40.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,369 [shard 0] osd - peering_event(id=772, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:40.374 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:56:40.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 423.003112793s (sent 423.003112793s) 2022-03-29T17:56:40.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - log is not dirty 2022-03-29T17:56:40.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - peering_event(id=771, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:40.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:40.375 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:56:40.376 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 423.003356934s (sent 423.003356934s) 2022-03-29T17:56:40.376 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - log is not dirty 2022-03-29T17:56:40.376 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - peering_event(id=773, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:40.376 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:40.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:56:40.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 423.003479004s (sent 423.003479004s) 2022-03-29T17:56:40.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - log is not dirty 2022-03-29T17:56:40.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - peering_event(id=772, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:40.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,370 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:40.378 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,371 [shard 0] osd - peering_event(id=771, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:40.378 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,371 [shard 0] osd - peering_event(id=773, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:40.378 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:40,371 [shard 0] osd - peering_event(id=772, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:40.703 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:40,702 [shard 0] alienstore - stat 2022-03-29T17:56:41.251 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:41.248+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:41.710 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:41,709 [shard 0] alienstore - stat 2022-03-29T17:56:41.993 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,990 [shard 0] osd - peering_event(id=774, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:41.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,991 [shard 0] osd - peering_event(id=774, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:41.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,991 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:56:41.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,991 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:56:41.994 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,991 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 424.624267578s) 2022-03-29T17:56:41.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,991 [shard 0] osd - log is not dirty 2022-03-29T17:56:41.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,991 [shard 0] osd - peering_event(id=774, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:41.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,991 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:41.995 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:41,992 [shard 0] osd - peering_event(id=774, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:42.003 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,000 [shard 0] osd - peering_event(id=775, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:42.004 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,001 [shard 0] osd - peering_event(id=775, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:42.004 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,001 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:56:42.004 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,001 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:56:42.004 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,001 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 424.634216309s) 2022-03-29T17:56:42.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,001 [shard 0] osd - log is not dirty 2022-03-29T17:56:42.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,001 [shard 0] osd - peering_event(id=775, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:42.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,001 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:42.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:42,001 [shard 0] osd - peering_event(id=775, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:42.446 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:42,443 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:42.446 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:42,443 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:42.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:42.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,443 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=13423821521847241135 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:42.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,443 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:42.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=11772863555078237645 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:42.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:42.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:42.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:42.448 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:42,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:42.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:42.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:42,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:42.646 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:42.647 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:42.648 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:42.649 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:42,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:42.649 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:42.650 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:42,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:42.717 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:42,716 [shard 0] alienstore - stat 2022-03-29T17:56:43.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,045 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:43.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:43.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:43.048 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:43,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:43.049 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:43.049 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:43,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:43.724 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,722 [shard 0] alienstore - stat 2022-03-29T17:56:43.848 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,846 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:43.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:43.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:43.849 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:43,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:43.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:43,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:43.850 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:43,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:44.290 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:44,288 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:44.291 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:44,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:56:44.291 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:44,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:44.291 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:44,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:56:44.292 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:44,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:56:44.730 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:44,729 [shard 0] alienstore - stat 2022-03-29T17:56:45.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2022-03-29T17:56:45.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~249s 2022-03-29T17:56:45.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:45,447 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:45.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:45,447 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:45.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:45,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:45.450 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:45,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:45.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:45,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:45.451 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:45,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:45.737 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:45,736 [shard 0] alienstore - stat 2022-03-29T17:56:46.251 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:46.249+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:46.744 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:46,742 [shard 0] alienstore - stat 2022-03-29T17:56:47.610 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,608 [shard 0] osd - peering_event(id=776, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:47.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,608 [shard 0] osd - peering_event(id=776, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:47.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,608 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:56:47.611 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,608 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:56:47.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,608 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 430.241729736s) 2022-03-29T17:56:47.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,608 [shard 0] osd - log is not dirty 2022-03-29T17:56:47.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,608 [shard 0] osd - peering_event(id=776, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:47.612 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,608 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:47.613 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,609 [shard 0] osd - peering_event(id=776, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:47.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,618 [shard 0] osd - peering_event(id=777, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:47.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,618 [shard 0] osd - peering_event(id=777, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:47.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,618 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:56:47.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,618 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:56:47.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,618 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 430.251708984s) 2022-03-29T17:56:47.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,618 [shard 0] osd - log is not dirty 2022-03-29T17:56:47.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,618 [shard 0] osd - peering_event(id=777, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:47.622 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,618 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:47.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,619 [shard 0] osd - peering_event(id=777, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:47.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,648 [shard 0] osd - peering_event(id=778, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:47.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,648 [shard 0] osd - peering_event(id=778, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:47.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,648 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:56:47.651 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,648 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:56:47.652 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,648 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 430.281799316s) 2022-03-29T17:56:47.652 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,648 [shard 0] osd - log is not dirty 2022-03-29T17:56:47.653 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,648 [shard 0] osd - peering_event(id=778, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:47.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,648 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:47.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,649 [shard 0] osd - peering_event(id=778, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:47.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,678 [shard 0] osd - peering_event(id=779, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:47.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,678 [shard 0] osd - peering_event(id=779, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:47.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,679 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:56:47.681 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,679 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:56:47.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,679 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 430.312072754s) 2022-03-29T17:56:47.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,679 [shard 0] osd - log is not dirty 2022-03-29T17:56:47.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,679 [shard 0] osd - peering_event(id=779, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:47.682 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,679 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:47.683 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,679 [shard 0] osd - peering_event(id=779, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:47.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,688 [shard 0] osd - peering_event(id=780, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:47.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,688 [shard 0] osd - peering_event(id=780, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:47.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,689 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:56:47.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,689 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:56:47.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,689 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 430.322113037s) 2022-03-29T17:56:47.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,689 [shard 0] osd - log is not dirty 2022-03-29T17:56:47.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,689 [shard 0] osd - peering_event(id=780, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:47.692 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,689 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:47.693 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,689 [shard 0] osd - peering_event(id=780, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:47.721 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,719 [shard 0] osd - peering_event(id=781, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:47.721 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,719 [shard 0] osd - peering_event(id=781, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:47.722 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,719 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:56:47.722 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,719 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:56:47.722 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,719 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 430.352325439s) 2022-03-29T17:56:47.722 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,719 [shard 0] osd - log is not dirty 2022-03-29T17:56:47.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,719 [shard 0] osd - peering_event(id=781, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:47.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,719 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:47.723 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:47,720 [shard 0] osd - peering_event(id=781, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:47.751 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:47,749 [shard 0] alienstore - stat 2022-03-29T17:56:47.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:47,874 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:56:47.877 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:47,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:56:47.877 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:47,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:47.877 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:47,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:56:47.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:47,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:56:48.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - peering_event(id=782, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:48.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - peering_event(id=783, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:48.357 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - peering_event(id=784, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:48.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - peering_event(id=782, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:48.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - peering_event(id=783, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:48.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - peering_event(id=784, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:48.358 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:56:48.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 430.986877441s (sent 430.986877441s) 2022-03-29T17:56:48.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - log is not dirty 2022-03-29T17:56:48.359 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - peering_event(id=782, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:48.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,353 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:48.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:56:48.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 430.987121582s (sent 430.987121582s) 2022-03-29T17:56:48.360 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - log is not dirty 2022-03-29T17:56:48.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - peering_event(id=783, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:48.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:48.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:56:48.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 430.987335205s (sent 430.987335205s) 2022-03-29T17:56:48.361 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - log is not dirty 2022-03-29T17:56:48.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - peering_event(id=784, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:48.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:48.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,354 [shard 0] osd - peering_event(id=782, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:48.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,355 [shard 0] osd - peering_event(id=783, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:48.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,355 [shard 0] osd - peering_event(id=784, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:48.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,373 [shard 0] osd - peering_event(id=785, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:48.377 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,373 [shard 0] osd - peering_event(id=786, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:48.378 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,373 [shard 0] osd - peering_event(id=787, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:48.378 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,373 [shard 0] osd - peering_event(id=785, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:48.378 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,373 [shard 0] osd - peering_event(id=787, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:48.378 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,373 [shard 0] osd - peering_event(id=786, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:48.379 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:56:48.379 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 431.007110596s (sent 431.007110596s) 2022-03-29T17:56:48.379 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - log is not dirty 2022-03-29T17:56:48.379 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - peering_event(id=787, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:48.380 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:48.380 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:56:48.380 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 431.007324219s (sent 431.007324219s) 2022-03-29T17:56:48.380 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - log is not dirty 2022-03-29T17:56:48.380 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - peering_event(id=785, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:48.381 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:48.381 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:56:48.381 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 431.007537842s (sent 431.007537842s) 2022-03-29T17:56:48.381 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - log is not dirty 2022-03-29T17:56:48.382 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - peering_event(id=786, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:48.382 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:48.382 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,374 [shard 0] osd - peering_event(id=787, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:48.382 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,375 [shard 0] osd - peering_event(id=785, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:48.382 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:48,375 [shard 0] osd - peering_event(id=786, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:48.446 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:48,443 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:48.447 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:48,444 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:48.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:48.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=15058609538206833445 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:48.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:48.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=11898548480133278931 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:48.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:48.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:48.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:48.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:48,445 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:48.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:48.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:48,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:48.647 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:48.647 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:48.648 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:48.648 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:48,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:48.648 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:48.648 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:48,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:48.757 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:48,756 [shard 0] alienstore - stat 2022-03-29T17:56:49.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,045 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:49.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:49.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:49.048 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:49,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:49.049 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:49.049 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:49,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:49.764 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,763 [shard 0] alienstore - stat 2022-03-29T17:56:49.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,846 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:49.851 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,846 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:49.851 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:49.852 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:49,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:49.852 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:49,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:49.852 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:49,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:49.996 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,994 [shard 0] osd - peering_event(id=788, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:49.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,994 [shard 0] osd - peering_event(id=788, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:49.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,994 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:56:49.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,994 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:56:49.997 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,994 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 432.627624512s) 2022-03-29T17:56:49.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,994 [shard 0] osd - log is not dirty 2022-03-29T17:56:49.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,994 [shard 0] osd - peering_event(id=788, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:49.998 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,994 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:49.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:49,995 [shard 0] osd - peering_event(id=788, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:50.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,004 [shard 0] osd - peering_event(id=789, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:50.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,004 [shard 0] osd - peering_event(id=789, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:50.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,004 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:56:50.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,004 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:56:50.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,004 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 432.637542725s) 2022-03-29T17:56:50.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,004 [shard 0] osd - log is not dirty 2022-03-29T17:56:50.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,004 [shard 0] osd - peering_event(id=789, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:50.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,004 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:50.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:50,005 [shard 0] osd - peering_event(id=789, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:50.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~282s 2022-03-29T17:56:50.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2022-03-29T17:56:50.771 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:50,769 [shard 0] alienstore - stat 2022-03-29T17:56:51.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:51.250+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:51.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:51,447 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:51.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:51,447 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:51.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:51,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:51.450 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:51,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:51.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:51,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:51.451 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:51,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:51.778 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:51,776 [shard 0] alienstore - stat 2022-03-29T17:56:52.785 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:52,783 [shard 0] alienstore - stat 2022-03-29T17:56:53.791 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:53,789 [shard 0] alienstore - stat 2022-03-29T17:56:54.446 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:56:54,443 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:56:54.447 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:54,444 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:56:54.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:56:54.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=11038123288447479464 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:54.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:56:54.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=155924102275646865 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:56:54.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:54.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:56:54.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,445 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:54.448 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:54,445 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:56:54.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:54.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:54,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:56:54.647 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:54.647 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:54.648 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:54.648 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:54,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:56:54.648 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,646 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:54.648 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:54,646 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:56:54.798 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:54,796 [shard 0] alienstore - stat 2022-03-29T17:56:55.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:55.048 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:55.049 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:55.049 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:55,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:56:55.049 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:55.049 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:55,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:56:55.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,611 [shard 0] osd - peering_event(id=790, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:55.614 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,612 [shard 0] osd - peering_event(id=790, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:55.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,612 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:56:55.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,612 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:56:55.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,612 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 438.245391846s) 2022-03-29T17:56:55.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,612 [shard 0] osd - log is not dirty 2022-03-29T17:56:55.615 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,612 [shard 0] osd - peering_event(id=790, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:55.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,612 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:55.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,613 [shard 0] osd - peering_event(id=790, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:55.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,621 [shard 0] osd - peering_event(id=791, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:55.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,622 [shard 0] osd - peering_event(id=791, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:55.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,622 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:56:55.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,622 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:56:55.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,622 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 438.255187988s) 2022-03-29T17:56:55.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,622 [shard 0] osd - log is not dirty 2022-03-29T17:56:55.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,622 [shard 0] osd - peering_event(id=791, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:55.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,622 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:55.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,622 [shard 0] osd - peering_event(id=791, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:55.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,652 [shard 0] osd - peering_event(id=792, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:55.654 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,652 [shard 0] osd - peering_event(id=792, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:55.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,652 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:56:55.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,652 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:56:55.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,652 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 438.285522461s) 2022-03-29T17:56:55.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,652 [shard 0] osd - log is not dirty 2022-03-29T17:56:55.655 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,652 [shard 0] osd - peering_event(id=792, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:55.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,652 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:55.656 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,653 [shard 0] osd - peering_event(id=792, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:55.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,682 [shard 0] osd - peering_event(id=793, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:55.686 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,682 [shard 0] osd - peering_event(id=793, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:55.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,682 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:56:55.687 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,682 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:56:55.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,682 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 438.315765381s) 2022-03-29T17:56:55.688 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,682 [shard 0] osd - log is not dirty 2022-03-29T17:56:55.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,682 [shard 0] osd - peering_event(id=793, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:55.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,682 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:55.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,683 [shard 0] osd - peering_event(id=793, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:55.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,692 [shard 0] osd - peering_event(id=794, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:55.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,692 [shard 0] osd - peering_event(id=794, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:55.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,692 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:56:55.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,692 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:56:55.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,692 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 438.325836182s) 2022-03-29T17:56:55.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,692 [shard 0] osd - log is not dirty 2022-03-29T17:56:55.698 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,692 [shard 0] osd - peering_event(id=794, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:55.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,692 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:55.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,693 [shard 0] osd - peering_event(id=794, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:55.725 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,722 [shard 0] osd - peering_event(id=795, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:56:55.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,722 [shard 0] osd - peering_event(id=795, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:56:55.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,722 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:56:55.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,722 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:56:55.726 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,723 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 438.355987549s) 2022-03-29T17:56:55.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,723 [shard 0] osd - log is not dirty 2022-03-29T17:56:55.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,723 [shard 0] osd - peering_event(id=795, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:56:55.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,723 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:55.727 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:55,723 [shard 0] osd - peering_event(id=795, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:56:55.805 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,803 [shard 0] alienstore - stat 2022-03-29T17:56:55.848 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:55.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:55.849 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:55.850 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:55,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:56:55.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:55,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:55.850 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:55,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:56:56.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~287s 2022-03-29T17:56:56.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2022-03-29T17:56:56.253 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:56:56.252+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:56:56.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,356 [shard 0] osd - peering_event(id=796, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:56.362 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,356 [shard 0] osd - peering_event(id=797, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:56.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - peering_event(id=798, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:56.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - peering_event(id=798, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:56.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - peering_event(id=797, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:56.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - peering_event(id=796, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:56.363 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:56:56.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 438.990478516s (sent 438.990478516s) 2022-03-29T17:56:56.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - log is not dirty 2022-03-29T17:56:56.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - peering_event(id=798, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:56.364 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:56.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:56:56.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 438.990722656s (sent 438.990722656s) 2022-03-29T17:56:56.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - log is not dirty 2022-03-29T17:56:56.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - peering_event(id=797, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:56.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:56.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:56:56.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 438.990966797s (sent 438.990966797s) 2022-03-29T17:56:56.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,357 [shard 0] osd - log is not dirty 2022-03-29T17:56:56.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,358 [shard 0] osd - peering_event(id=796, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:56.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,358 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:56.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,358 [shard 0] osd - peering_event(id=798, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:56.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,358 [shard 0] osd - peering_event(id=797, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:56.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,358 [shard 0] osd - peering_event(id=796, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:56.382 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,376 [shard 0] osd - peering_event(id=799, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:56.383 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,376 [shard 0] osd - peering_event(id=800, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:56.383 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,376 [shard 0] osd - peering_event(id=801, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:56:56.383 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - peering_event(id=800, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:56.383 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - peering_event(id=799, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:56.384 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - peering_event(id=801, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:56:56.384 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:56:56.384 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 439.010284424s (sent 439.010284424s) 2022-03-29T17:56:56.384 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - log is not dirty 2022-03-29T17:56:56.384 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - peering_event(id=800, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:56.385 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:56.385 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:56:56.385 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 439.010528564s (sent 439.010528564s) 2022-03-29T17:56:56.385 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - log is not dirty 2022-03-29T17:56:56.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - peering_event(id=801, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:56.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:56.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:56:56.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 439.010742188s (sent 439.010742188s) 2022-03-29T17:56:56.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - log is not dirty 2022-03-29T17:56:56.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - peering_event(id=799, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:56:56.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,377 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:56.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,378 [shard 0] osd - peering_event(id=800, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:56.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,378 [shard 0] osd - peering_event(id=799, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:56.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,379 [shard 0] osd - peering_event(id=801, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:56:56.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,766 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4127.0:14 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): start 2022-03-29T17:56:56.772 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,766 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4127.0:14 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): in repeat 2022-03-29T17:56:56.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,766 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4127.0:14 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]) same_interval_since: 11 2022-03-29T17:56:56.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - OpSequencer::start_op: op=15, last_started=13, last_unblocked=13, last_completed=13 2022-03-29T17:56:56.773 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - do_recover_missing check for recovery, 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:56:56.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4127.0:14 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): got obc lock 2022-03-29T17:56:56.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - with_head_obc 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:56:56.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - with_head_obc: found 2:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:56:56.774 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:56:56.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - handling op omap-get-vals on object 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:56:56.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - do_osd_ops_execute: object 2:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:56:56.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,767 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:14 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8 - object 2:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:56:56.775 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - with_head_obc: released 2:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:56:56.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - OpSequencer::finish_op_in_order: op=15, last_started=15, last_unblocked=15, last_completed=13 2022-03-29T17:56:56.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - client_request(id=15, detail=m=[osd_op(client.4127.0:14 2.1 2:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): destroying 2022-03-29T17:56:56.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4127.0:15 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): start 2022-03-29T17:56:56.776 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4127.0:15 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): in repeat 2022-03-29T17:56:56.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4127.0:15 3.11 3.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e20) v8]) same_interval_since: 14 2022-03-29T17:56:56.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - OpSequencer::start_op: op=16, last_started=14, last_unblocked=14, last_completed=14 2022-03-29T17:56:56.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - do_recover_missing check for recovery, 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:56:56.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,768 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4127.0:15 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): got obc lock 2022-03-29T17:56:56.777 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - with_head_obc 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:56:56.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - with_head_obc: found 3:88c1567c:::rbd_trash_purge_schedule:head in cache 2022-03-29T17:56:56.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head - handling op omap-get-vals 2022-03-29T17:56:56.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - handling op omap-get-vals on object 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:56:56.778 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - do_osd_ops_execute: object 3:88c1567c:::rbd_trash_purge_schedule:head all operations successful 2022-03-29T17:56:56.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - do_osd_ops: osd_op(client.4127.0:15 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8 - object 3:88c1567c:::rbd_trash_purge_schedule:head sending reply 2022-03-29T17:56:56.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - with_head_obc: released 3:88c1567c:::rbd_trash_purge_schedule:head 2022-03-29T17:56:56.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - OpSequencer::finish_op_in_order: op=16, last_started=16, last_unblocked=16, last_completed=14 2022-03-29T17:56:56.779 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:56,769 [shard 0] osd - client_request(id=16, detail=m=[osd_op(client.4127.0:15 3.11 3:88c1567c:::rbd_trash_purge_schedule:head {omap-get-vals in=16b out=5b} snapc 0={} ondisk+read+known_if_redirected+supports_pool_eio e20) v8]): destroying 2022-03-29T17:56:56.811 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:56,809 [shard 0] alienstore - stat 2022-03-29T17:56:57.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:57,447 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:57.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:57,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:57.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:57,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:57.451 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:57,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:56:57.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:57,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:57.451 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:57,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:56:57.818 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:57,816 [shard 0] alienstore - stat 2022-03-29T17:56:57.999 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,997 [shard 0] osd - peering_event(id=802, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:58.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,997 [shard 0] osd - peering_event(id=802, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:58.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,997 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:56:58.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,997 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:56:58.000 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,997 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 440.630706787s) 2022-03-29T17:56:58.001 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,997 [shard 0] osd - log is not dirty 2022-03-29T17:56:58.001 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,997 [shard 0] osd - peering_event(id=802, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:58.001 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,997 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:58.002 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:57,998 [shard 0] osd - peering_event(id=802, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:58.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,007 [shard 0] osd - peering_event(id=803, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:56:58.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,007 [shard 0] osd - peering_event(id=803, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:56:58.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,007 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:56:58.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,007 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:56:58.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,007 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 440.640747070s) 2022-03-29T17:56:58.010 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,007 [shard 0] osd - log is not dirty 2022-03-29T17:56:58.011 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,007 [shard 0] osd - peering_event(id=803, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:56:58.011 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,007 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:56:58.011 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:56:58,008 [shard 0] osd - peering_event(id=803, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:56:58.824 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:58,823 [shard 0] alienstore - stat 2022-03-29T17:56:59.291 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:59,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:56:59.291 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:59,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:56:59.291 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:59,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:56:59.292 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:56:59,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:56:59.292 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:59,289 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:56:59.831 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:56:59,830 [shard 0] alienstore - stat 2022-03-29T17:57:00.447 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:57:00,444 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:57:00.447 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:00,444 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:57:00.447 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:57:00.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=14495233846274704680 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:00.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:57:00.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=12032153726164543165 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:00.448 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,444 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:00.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,444 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:00.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:00.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:00,445 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:57:00.449 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,445 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:00.449 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:00,445 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:57:00.647 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,645 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:00.648 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,645 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:00.648 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,646 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:00.648 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:00,646 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:57:00.649 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,646 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:00.649 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:00,646 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:57:00.838 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:00,837 [shard 0] alienstore - stat 2022-03-29T17:57:01.049 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,046 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:01.049 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,046 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:01.050 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,047 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:01.050 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:01,047 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:57:01.050 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,047 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:01.050 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:01,047 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:57:01.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:57:01.253+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:57:01.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~292s 2022-03-29T17:57:01.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2022-03-29T17:57:01.845 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,844 [shard 0] alienstore - stat 2022-03-29T17:57:01.848 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:01.848 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,847 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:01.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:01.850 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:01,847 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:57:01.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:01,848 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:01.850 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:01,848 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:57:02.852 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:02,851 [shard 0] alienstore - stat 2022-03-29T17:57:02.877 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:02,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): going to CONNECTING 2022-03-29T17:57:02.877 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:02,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:57:02.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:02,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:02.878 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:02,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] waiting 15.0 seconds ... 2022-03-29T17:57:02.878 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:02,875 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: delay ... 2022-03-29T17:57:03.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:03,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:03.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:03,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:03.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:03,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:03.451 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:03,448 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:57:03.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:03,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:03.451 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:03,448 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:57:03.618 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,616 [shard 0] osd - peering_event(id=804, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:03.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,616 [shard 0] osd - peering_event(id=804, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:03.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,616 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:57:03.619 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,616 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:57:03.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,616 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 446.249664307s) 2022-03-29T17:57:03.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,616 [shard 0] osd - log is not dirty 2022-03-29T17:57:03.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,616 [shard 0] osd - peering_event(id=804, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:03.620 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,616 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:03.621 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,617 [shard 0] osd - peering_event(id=804, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:03.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,626 [shard 0] osd - peering_event(id=805, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:03.628 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,626 [shard 0] osd - peering_event(id=805, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:03.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,626 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:57:03.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,626 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:57:03.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,626 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 446.259521484s) 2022-03-29T17:57:03.629 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,626 [shard 0] osd - log is not dirty 2022-03-29T17:57:03.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,626 [shard 0] osd - peering_event(id=805, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:03.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,626 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:03.630 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,627 [shard 0] osd - peering_event(id=805, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:03.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,656 [shard 0] osd - peering_event(id=806, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:03.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,656 [shard 0] osd - peering_event(id=806, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:03.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,656 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:57:03.659 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,656 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:57:03.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,656 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 446.289733887s) 2022-03-29T17:57:03.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,656 [shard 0] osd - log is not dirty 2022-03-29T17:57:03.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,656 [shard 0] osd - peering_event(id=806, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:03.660 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,656 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:03.661 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,657 [shard 0] osd - peering_event(id=806, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:03.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,686 [shard 0] osd - peering_event(id=807, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:03.689 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,686 [shard 0] osd - peering_event(id=807, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:03.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,686 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:57:03.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,686 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:57:03.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,687 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 446.319976807s) 2022-03-29T17:57:03.690 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,687 [shard 0] osd - log is not dirty 2022-03-29T17:57:03.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,687 [shard 0] osd - peering_event(id=807, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:03.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,687 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:03.691 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,687 [shard 0] osd - peering_event(id=807, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:03.699 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,696 [shard 0] osd - peering_event(id=808, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:03.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,696 [shard 0] osd - peering_event(id=808, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:03.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,697 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:57:03.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,697 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:57:03.700 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,697 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 446.330139160s) 2022-03-29T17:57:03.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,697 [shard 0] osd - log is not dirty 2022-03-29T17:57:03.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,697 [shard 0] osd - peering_event(id=808, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:03.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,697 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:03.701 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,697 [shard 0] osd - peering_event(id=808, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:03.729 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,727 [shard 0] osd - peering_event(id=809, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:03.729 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,727 [shard 0] osd - peering_event(id=809, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:03.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,727 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:57:03.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,727 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:57:03.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,727 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 446.360290527s) 2022-03-29T17:57:03.730 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,727 [shard 0] osd - log is not dirty 2022-03-29T17:57:03.731 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,727 [shard 0] osd - peering_event(id=809, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:03.731 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,727 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:03.731 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:03,728 [shard 0] osd - peering_event(id=809, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:03.859 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:03,858 [shard 0] alienstore - stat 2022-03-29T17:57:04.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - peering_event(id=810, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:04.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - peering_event(id=811, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:04.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - peering_event(id=812, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:04.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - peering_event(id=812, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:04.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - peering_event(id=811, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:04.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - peering_event(id=810, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:04.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:57:04.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 446.994781494s (sent 446.994781494s) 2022-03-29T17:57:04.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - log is not dirty 2022-03-29T17:57:04.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - peering_event(id=812, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:04.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:04.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,361 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:57:04.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 446.995025635s (sent 446.995025635s) 2022-03-29T17:57:04.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - log is not dirty 2022-03-29T17:57:04.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - peering_event(id=810, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:04.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:04.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:57:04.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 446.995239258s (sent 446.995239258s) 2022-03-29T17:57:04.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - log is not dirty 2022-03-29T17:57:04.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - peering_event(id=811, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:04.372 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:04.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - peering_event(id=812, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:04.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - peering_event(id=810, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:04.373 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,362 [shard 0] osd - peering_event(id=811, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:04.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - peering_event(id=813, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:04.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - peering_event(id=814, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:04.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - peering_event(id=815, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:04.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - peering_event(id=813, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:04.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - peering_event(id=815, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:04.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - peering_event(id=814, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:04.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:57:04.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 447.014862061s (sent 447.014862061s) 2022-03-29T17:57:04.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - log is not dirty 2022-03-29T17:57:04.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - peering_event(id=814, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:04.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,381 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:04.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:57:04.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 447.015106201s (sent 447.015106201s) 2022-03-29T17:57:04.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - log is not dirty 2022-03-29T17:57:04.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - peering_event(id=815, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:04.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:04.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:57:04.391 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 447.015319824s (sent 447.015319824s) 2022-03-29T17:57:04.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - log is not dirty 2022-03-29T17:57:04.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - peering_event(id=813, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:04.392 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:04.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,382 [shard 0] osd - peering_event(id=814, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:04.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,383 [shard 0] osd - peering_event(id=813, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:04.393 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:04,383 [shard 0] osd - peering_event(id=815, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:04.866 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:04,864 [shard 0] alienstore - stat 2022-03-29T17:57:05.873 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:05,872 [shard 0] alienstore - stat 2022-03-29T17:57:06.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,002 [shard 0] osd - peering_event(id=816, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:57:06.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,002 [shard 0] osd - peering_event(id=816, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:57:06.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,002 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:57:06.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,002 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:57:06.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,002 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 448.635925293s) 2022-03-29T17:57:06.008 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,003 [shard 0] osd - log is not dirty 2022-03-29T17:57:06.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,003 [shard 0] osd - peering_event(id=816, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:57:06.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,003 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:06.009 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,003 [shard 0] osd - peering_event(id=816, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:57:06.016 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,012 [shard 0] osd - peering_event(id=817, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:57:06.016 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,012 [shard 0] osd - peering_event(id=817, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:57:06.017 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,013 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:57:06.017 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,013 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:57:06.017 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,013 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 448.646087646s) 2022-03-29T17:57:06.018 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,013 [shard 0] osd - log is not dirty 2022-03-29T17:57:06.018 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,013 [shard 0] osd - peering_event(id=817, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:57:06.019 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,013 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:06.019 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:06,013 [shard 0] osd - peering_event(id=817, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:57:06.256 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:57:06.252+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:57:06.449 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:57:06,445 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:57:06.450 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:06,445 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:57:06.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:57:06.450 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=12547276732195135172 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:06.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,446 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:57:06.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,446 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=2356789630279932106 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:06.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,446 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:06.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,446 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:06.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,447 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:06.452 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:06,447 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:57:06.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,447 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:06.452 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:06,447 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:57:06.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~298s 2022-03-29T17:57:06.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2022-03-29T17:57:06.650 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,647 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:06.650 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,647 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:06.651 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,647 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:06.651 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:06,647 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:57:06.652 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,647 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:06.652 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:06,648 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:57:06.880 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:06,878 [shard 0] alienstore - stat 2022-03-29T17:57:07.050 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,048 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:07.050 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,048 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:07.050 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,048 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:07.050 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:07,048 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:57:07.051 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,048 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:07.051 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:07,048 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:57:07.850 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,848 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:07.851 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,848 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:07.851 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,849 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:07.851 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:07,849 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:57:07.852 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,849 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:07.852 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:07,849 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:57:07.887 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:07,885 [shard 0] alienstore - stat 2022-03-29T17:57:08.894 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:08,892 [shard 0] alienstore - stat 2022-03-29T17:57:09.451 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:09,449 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:09.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:09,449 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:09.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:09,450 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:09.452 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:09,450 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:57:09.452 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:09,450 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:09.453 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:09,450 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:57:09.900 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:09,899 [shard 0] alienstore - stat 2022-03-29T17:57:10.907 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:10,905 [shard 0] alienstore - stat 2022-03-29T17:57:11.256 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:57:11.253+0000 7fcedb7fe700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.138:6800/35578 for 30sec e20 v20) 2022-03-29T17:57:11.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,620 [shard 0] osd - peering_event(id=818, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:11.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,620 [shard 0] osd - peering_event(id=818, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:11.623 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,621 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.8 2022-03-29T17:57:11.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,621 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.123367310s 2022-03-29T17:57:11.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,621 [shard 0] osd - pg_epoch 20 pg[3.8( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.123367310s (sent 454.254211426s) 2022-03-29T17:57:11.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,621 [shard 0] osd - log is not dirty 2022-03-29T17:57:11.624 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,621 [shard 0] osd - peering_event(id=818, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:11.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,621 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:11.625 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,622 [shard 0] osd - peering_event(id=818, detail=PeeringEvent(from=2 pgid=3.8 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:11.632 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,630 [shard 0] osd - peering_event(id=819, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:11.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,630 [shard 0] osd - peering_event(id=819, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:11.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,631 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.7 2022-03-29T17:57:11.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,631 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until peer osd.1 ruub 174.133697510s 2022-03-29T17:57:11.633 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,631 [shard 0] osd - pg_epoch 20 pg[3.7( v 18'1 (0'0,18'1] local-lis/les=14/15 n=1 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 lua=0'0 crt=18'1 mlcod 18'1 active+clean recalc_readable_until readable_until[_ub] 174.133697510s (sent 454.264129639s) 2022-03-29T17:57:11.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,631 [shard 0] osd - log is not dirty 2022-03-29T17:57:11.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,631 [shard 0] osd - peering_event(id=819, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:11.634 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,631 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:11.635 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,631 [shard 0] osd - peering_event(id=819, detail=PeeringEvent(from=2 pgid=3.7 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:11.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,660 [shard 0] osd - peering_event(id=820, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:11.665 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,661 [shard 0] osd - peering_event(id=820, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:11.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,661 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.1e 2022-03-29T17:57:11.666 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,661 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.163772583s 2022-03-29T17:57:11.667 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,661 [shard 0] osd - pg_epoch 20 pg[3.1e( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.163772583s (sent 454.294189453s) 2022-03-29T17:57:11.667 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,661 [shard 0] osd - log is not dirty 2022-03-29T17:57:11.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,661 [shard 0] osd - peering_event(id=820, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:11.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,661 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:11.668 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,661 [shard 0] osd - peering_event(id=820, detail=PeeringEvent(from=2 pgid=3.1e sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:11.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,691 [shard 0] osd - peering_event(id=821, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:11.694 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,691 [shard 0] osd - peering_event(id=821, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:11.695 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,691 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.16 2022-03-29T17:57:11.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,691 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.193771362s 2022-03-29T17:57:11.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,691 [shard 0] osd - pg_epoch 20 pg[3.16( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.193771362s (sent 454.324310303s) 2022-03-29T17:57:11.696 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,691 [shard 0] osd - log is not dirty 2022-03-29T17:57:11.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,691 [shard 0] osd - peering_event(id=821, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:11.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,691 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:11.697 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,692 [shard 0] osd - peering_event(id=821, detail=PeeringEvent(from=2 pgid=3.16 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:11.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,701 [shard 0] osd - peering_event(id=822, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:11.703 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,701 [shard 0] osd - peering_event(id=822, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:11.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,701 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.5 2022-03-29T17:57:11.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,701 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.203826904s 2022-03-29T17:57:11.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,701 [shard 0] osd - pg_epoch 20 pg[3.5( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.203826904s (sent 454.334411621s) 2022-03-29T17:57:11.704 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,701 [shard 0] osd - log is not dirty 2022-03-29T17:57:11.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,701 [shard 0] osd - peering_event(id=822, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:11.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,701 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:11.705 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,702 [shard 0] osd - peering_event(id=822, detail=PeeringEvent(from=2 pgid=3.5 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:11.733 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,731 [shard 0] osd - peering_event(id=823, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): start 2022-03-29T17:57:11.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,731 [shard 0] osd - peering_event(id=823, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): pg present 2022-03-29T17:57:11.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,731 [shard 0] osd - do_peering_event handling epoch_sent: 14 epoch_requested: 14 RenewLease for pg: 3.11 2022-03-29T17:57:11.734 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,731 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 174.234085083s 2022-03-29T17:57:11.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,731 [shard 0] osd - pg_epoch 20 pg[3.11( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [2,1] r=0 lpr=14 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 174.234085083s (sent 454.364685059s) 2022-03-29T17:57:11.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,731 [shard 0] osd - log is not dirty 2022-03-29T17:57:11.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,731 [shard 0] osd - peering_event(id=823, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): submitting ctx 2022-03-29T17:57:11.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,731 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:11.735 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:11,732 [shard 0] osd - peering_event(id=823, detail=PeeringEvent(from=2 pgid=3.11 sent=14 requested=14 evt=epoch_sent: 14 epoch_requested: 14 RenewLease)): complete 2022-03-29T17:57:11.914 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:11,912 [shard 0] alienstore - stat 2022-03-29T17:57:11.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~303s 2022-03-29T17:57:11.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2022-03-29T17:57:11.948 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-03-29T17:57:12.251 INFO:tasks.ceph.osd.2:Sent signal 15 2022-03-29T17:57:12.253 INFO:tasks.ceph.mon.a:Sent signal 15 2022-03-29T17:57:12.253 INFO:tasks.ceph.mgr.x:Sent signal 15 2022-03-29T17:57:12.254 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,250 [shard 0] osd - crimson shutting down 2022-03-29T17:57:12.255 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,250 [shard 0] osd - stop 2022-03-29T17:57:12.255 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:57:12.251+0000 7fceea968700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 35188) UID: 0 2022-03-29T17:57:12.255 INFO:tasks.ceph.mon.a.smithi138.stderr:2022-03-29T17:57:12.251+0000 7fceea968700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-03-29T17:57:12.256 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:57:12.252+0000 7fb01949e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 35190) UID: 0 2022-03-29T17:57:12.256 INFO:tasks.ceph.mgr.x.smithi138.stderr:2022-03-29T17:57:12.252+0000 7fb01949e700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-03-29T17:57:12.262 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,260 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@56603 >> mon.0 v2:172.21.15.138:3300/0] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:57:12.263 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,260 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@56603 >> mon.0 v2:172.21.15.138:3300/0] closing: reset yes, replace no 2022-03-29T17:57:12.263 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,261 [shard 0] osd - ms_handle_reset 2022-03-29T17:57:12.263 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,261 [shard 0] monc - active conn reset v2:172.21.15.138:3300/0 2022-03-29T17:57:12.263 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,261 [shard 0] monc - reopen_session to mon.-1 2022-03-29T17:57:12.264 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,261 [shard 0] monc - close 2022-03-29T17:57:12.264 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,261 [shard 0] monc - connecting to mon.0 2022-03-29T17:57:12.264 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,261 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:3300/0, peer_name=mon.?, cc=315767057868166539 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:12.264 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,261 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:12.265 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,262 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:12.265 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,262 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 0.2 seconds ... 2022-03-29T17:57:12.266 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,262 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:12.274 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,272 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4127 172.21.15.138:0/3450850499@41188] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:57:12.275 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,272 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4127 172.21.15.138:0/3450850499@41188] closing: reset yes, replace no 2022-03-29T17:57:12.275 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,272 [shard 0] osd - ms_handle_reset 2022-03-29T17:57:12.275 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,272 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.4102 172.21.15.138:0/35195@41076] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:57:12.275 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,272 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.4102 172.21.15.138:0/35195@41076] closing: reset yes, replace no 2022-03-29T17:57:12.276 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,272 [shard 0] osd - ms_handle_reset 2022-03-29T17:57:12.276 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,273 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@54975 >> mgr.4102 v2:172.21.15.138:6812/35195] execute_ready(): fault at READY on lossy channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:57:12.276 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,273 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579@54975 >> mgr.4102 v2:172.21.15.138:6812/35195] closing: reset yes, replace no 2022-03-29T17:57:12.276 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,273 [shard 0] osd - ms_handle_reset 2022-03-29T17:57:12.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - peering_event(id=824, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:12.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - peering_event(id=825, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:12.365 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - peering_event(id=826, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:12.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - peering_event(id=825, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:12.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - peering_event(id=824, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:12.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - peering_event(id=826, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:12.366 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1d 2022-03-29T17:57:12.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - pg_epoch 20 pg[3.1d( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 454.994812012s (sent 454.994812012s) 2022-03-29T17:57:12.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - log is not dirty 2022-03-29T17:57:12.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - peering_event(id=825, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:12.367 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,361 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:12.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.e 2022-03-29T17:57:12.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - pg_epoch 20 pg[3.e( v 18'1 (0'0,18'1] local-lis/les=19/20 n=1 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=18'1 mlcod 0'0 active+undersized+degraded recalc_readable_until readable_until[_ub] 454.995117188s (sent 454.995117188s) 2022-03-29T17:57:12.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - log is not dirty 2022-03-29T17:57:12.368 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - peering_event(id=826, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:12.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:12.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.18 2022-03-29T17:57:12.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - pg_epoch 20 pg[3.18( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 454.995330811s (sent 454.995330811s) 2022-03-29T17:57:12.369 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - log is not dirty 2022-03-29T17:57:12.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - peering_event(id=824, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:12.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:12.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,362 [shard 0] osd - peering_event(id=825, detail=PeeringEvent(from=2 pgid=3.1d sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:12.370 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,363 [shard 0] osd - peering_event(id=826, detail=PeeringEvent(from=2 pgid=3.e sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:12.371 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,363 [shard 0] osd - peering_event(id=824, detail=PeeringEvent(from=2 pgid=3.18 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:12.385 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - peering_event(id=827, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:12.385 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - peering_event(id=828, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:12.385 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - peering_event(id=829, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): start 2022-03-29T17:57:12.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - peering_event(id=828, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:12.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - peering_event(id=829, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:12.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - peering_event(id=827, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): pg present 2022-03-29T17:57:12.386 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.3 2022-03-29T17:57:12.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - pg_epoch 20 pg[3.3( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 455.014801025s (sent 455.014801025s) 2022-03-29T17:57:12.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - log is not dirty 2022-03-29T17:57:12.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - peering_event(id=827, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:12.387 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:12.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.1 2022-03-29T17:57:12.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - pg_epoch 20 pg[3.1( empty local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=0'0 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 455.014984131s (sent 455.014984131s) 2022-03-29T17:57:12.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - log is not dirty 2022-03-29T17:57:12.388 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,381 [shard 0] osd - peering_event(id=828, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:12.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,382 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:12.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,382 [shard 0] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 3.9 2022-03-29T17:57:12.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,382 [shard 0] osd - pg_epoch 20 pg[3.9( v 17'1 (0'0,17'1] local-lis/les=19/20 n=0 ec=14/14 lis/c=19/14 les/c/f=20/15/0 sis=19) [2] r=0 lpr=19 pi=[14,19)/1 crt=17'1 mlcod 0'0 active+undersized recalc_readable_until readable_until[_ub] 455.015106201s (sent 455.015106201s) 2022-03-29T17:57:12.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,382 [shard 0] osd - log is not dirty 2022-03-29T17:57:12.389 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,382 [shard 0] osd - peering_event(id=829, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): submitting ctx 2022-03-29T17:57:12.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,382 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:12.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,382 [shard 0] osd - peering_event(id=828, detail=PeeringEvent(from=2 pgid=3.1 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:12.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,382 [shard 0] osd - peering_event(id=829, detail=PeeringEvent(from=2 pgid=3.9 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:12.390 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:12,383 [shard 0] osd - peering_event(id=827, detail=PeeringEvent(from=2 pgid=3.3 sent=19 requested=19 evt=epoch_sent: 19 epoch_requested: 19 RenewLease)): complete 2022-03-29T17:57:12.454 INFO:tasks.ceph.osd.2.smithi138.stderr:ERROR 2022-03-29 17:57:12,451 [shard 0] osd - Heartbeat::Session::failed_since(): no reply from osd.1 since back 2022-03-29 17:52:30 359757596 front 2022-03-29 17:52:30 359757596 (oldest deadline 2022-03-29 17:52:56 358634371) 2022-03-29T17:57:12.454 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,451 [shard 0] osd - Heartbeat::Peer::send_heartbeat(): heartbeat to osd.1 is still pending... 2022-03-29T17:57:12.455 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,451 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:57:12.455 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,452 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6808/35578, peer_name=osd.1, cc=11991269863795378072 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:12.455 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,452 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:57:12.455 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,452 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6807/35578, peer_name=osd.1, cc=5268757359142247775 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:12.456 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,452 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:12.456 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,452 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:12.456 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,452 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:12.456 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,452 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.2 seconds ... 2022-03-29T17:57:12.456 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,453 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:12.457 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,453 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.2 seconds ... 2022-03-29T17:57:12.473 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,462 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): going to CONNECTING 2022-03-29T17:57:12.473 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,462 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:12.474 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,463 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:12.474 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,463 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 0.4 seconds ... 2022-03-29T17:57:12.474 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,463 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:12.654 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,653 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:12.655 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,653 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:12.656 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,653 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:12.656 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,653 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.4 seconds ... 2022-03-29T17:57:12.656 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,653 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:12.656 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,653 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.4 seconds ... 2022-03-29T17:57:12.865 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,863 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): going to CONNECTING 2022-03-29T17:57:12.865 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,863 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:12.866 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,863 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:12.866 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:12,863 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 0.8 seconds ... 2022-03-29T17:57:12.866 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:12,863 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:13.056 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,053 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:13.056 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,054 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:13.056 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,054 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:13.056 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:13,054 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 0.8 seconds ... 2022-03-29T17:57:13.057 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,054 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:13.057 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:13,054 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 0.8 seconds ... 2022-03-29T17:57:13.275 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,273 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:6812/35195, peer_name=mgr.?, cc=3532257918776133239 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:13.275 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,273 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:13.276 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:13,273 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] waiting 0.2 seconds ... 2022-03-29T17:57:13.476 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,474 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_wait(): going to CONNECTING 2022-03-29T17:57:13.476 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,474 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:13.476 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:13,474 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] waiting 0.4 seconds ... 2022-03-29T17:57:13.666 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,663 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): going to CONNECTING 2022-03-29T17:57:13.667 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,664 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:13.667 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,664 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:13.668 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:13,664 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 1.6 seconds ... 2022-03-29T17:57:13.668 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,664 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:13.856 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,854 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:13.857 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,854 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:13.857 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,855 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:13.857 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:13,855 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 1.6 seconds ... 2022-03-29T17:57:13.858 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,855 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:13.858 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:13,855 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 1.6 seconds ... 2022-03-29T17:57:13.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,874 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_wait(): going to CONNECTING 2022-03-29T17:57:13.876 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:13,875 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:13.877 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:13,875 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] waiting 0.8 seconds ... 2022-03-29T17:57:14.004 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,002 [shard 0] osd - peering_event(id=830, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:57:14.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,002 [shard 0] osd - peering_event(id=830, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:57:14.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,002 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.0 2022-03-29T17:57:14.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,002 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.503524780s 2022-03-29T17:57:14.005 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,002 [shard 0] osd - pg_epoch 20 pg[2.0( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.503524780s (sent 456.635864258s) 2022-03-29T17:57:14.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,002 [shard 0] osd - log is not dirty 2022-03-29T17:57:14.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,003 [shard 0] osd - peering_event(id=830, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:57:14.006 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,003 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:14.007 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,003 [shard 0] osd - peering_event(id=830, detail=PeeringEvent(from=2 pgid=2.0 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:57:14.014 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,012 [shard 0] osd - peering_event(id=831, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): start 2022-03-29T17:57:14.014 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,012 [shard 0] osd - peering_event(id=831, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): pg present 2022-03-29T17:57:14.015 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,012 [shard 0] osd - do_peering_event handling epoch_sent: 11 epoch_requested: 11 RenewLease for pg: 2.1 2022-03-29T17:57:14.015 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,012 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until peer osd.1 ruub 168.513504028s 2022-03-29T17:57:14.015 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,012 [shard 0] osd - pg_epoch 20 pg[2.1( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [2,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean recalc_readable_until readable_until[_ub] 168.513504028s (sent 456.645751953s) 2022-03-29T17:57:14.015 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,012 [shard 0] osd - log is not dirty 2022-03-29T17:57:14.016 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,012 [shard 0] osd - peering_event(id=831, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): submitting ctx 2022-03-29T17:57:14.016 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,012 [shard 0] osd - ShardServices::dispatch_context_transaction: do_transaction... 2022-03-29T17:57:14.016 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,013 [shard 0] osd - peering_event(id=831, detail=PeeringEvent(from=2 pgid=2.1 sent=11 requested=11 evt=epoch_sent: 11 epoch_requested: 11 RenewLease)): complete 2022-03-29T17:57:14.206 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,203 [shard 0] monc - operator() continuing the hunt 2022-03-29T17:57:14.207 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,203 [shard 0] monc - reopen_session to mon.-1 2022-03-29T17:57:14.207 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,203 [shard 0] monc - close 2022-03-29T17:57:14.207 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,203 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] closing: reset no, replace no 2022-03-29T17:57:14.207 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,204 [shard 0] monc - connecting to mon.0 2022-03-29T17:57:14.207 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,204 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.138:3300/0, peer_name=mon.?, cc=6732951875873380295 policy(lossy=true, server=false, standby=false, resetcheck=false) 2022-03-29T17:57:14.208 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,204 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:14.208 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,204 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:14.208 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,204 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: dropped 2022-03-29T17:57:14.208 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:14,204 [shard 0] monc - reopen_session mon connection attempts complete 2022-03-29T17:57:14.209 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:14,204 [shard 0] monc - cannot establish the active_con with any mon 2022-03-29T17:57:14.209 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,204 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:14.209 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:14,204 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 0.2 seconds ... 2022-03-29T17:57:14.209 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,204 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:14.292 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,290 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:14.293 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,290 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:57:14.293 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,290 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:14.294 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:14,290 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] waiting 15.0 seconds ... 2022-03-29T17:57:14.294 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,290 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: delay ... 2022-03-29T17:57:14.407 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,404 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): going to CONNECTING 2022-03-29T17:57:14.409 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,405 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:14.409 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,405 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:14.409 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:14,405 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 0.4 seconds ... 2022-03-29T17:57:14.410 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,405 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:14.677 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,675 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_wait(): going to CONNECTING 2022-03-29T17:57:14.678 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,675 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:14.678 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:14,675 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] waiting 1.6 seconds ... 2022-03-29T17:57:14.807 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,805 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): going to CONNECTING 2022-03-29T17:57:14.807 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,805 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:14.808 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,805 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:14.808 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:14,805 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 0.8 seconds ... 2022-03-29T17:57:14.808 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:14,805 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:15.458 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:15,455 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:15.458 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:15,455 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): going to CONNECTING 2022-03-29T17:57:15.458 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:15,455 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:15.458 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:15,456 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] waiting 3.2 seconds ... 2022-03-29T17:57:15.459 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:15,456 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:15.459 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:15,456 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] waiting 3.2 seconds ... 2022-03-29T17:57:15.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:15,605 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): going to CONNECTING 2022-03-29T17:57:15.610 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:15,606 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:15.611 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:15,606 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:15.612 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:15,606 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 1.6 seconds ... 2022-03-29T17:57:15.612 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:15,606 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:16.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:16,275 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_wait(): going to CONNECTING 2022-03-29T17:57:16.277 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:16,276 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:16.278 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:16,276 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] waiting 3.2 seconds ... 2022-03-29T17:57:17.208 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,206 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): going to CONNECTING 2022-03-29T17:57:17.209 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,206 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:17.209 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,206 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_connecting(): fault at CONNECTING, going to WAIT -- std::system_error (error system:111, Connection refused) 2022-03-29T17:57:17.209 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:17,207 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] waiting 3.2 seconds ... 2022-03-29T17:57:17.210 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,207 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: delay ... 2022-03-29T17:57:17.254 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:17,250 [shard 0] osd - prepared to stop 2022-03-29T17:57:17.254 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,251 [shard 0] osd - AdminSocket: stopped 2022-03-29T17:57:17.255 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,251 [shard 0] osd - stop 2022-03-29T17:57:17.255 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,251 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] closing: reset no, replace no 2022-03-29T17:57:17.256 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,251 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] closing: reset no, replace no 2022-03-29T17:57:17.256 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,251 [shard 0] ms - [osd.2(hb_front) v2:172.21.15.138:6810/35579 >> osd.1 v2:172.21.15.138:6808/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:17.257 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,251 [shard 0] ms - [osd.2(hb_back) v2:172.21.15.138:6811/35579 >> osd.1 v2:172.21.15.138:6807/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:17.258 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,252 [shard 0] alienstore - umount 2022-03-29T17:57:17.599 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,587 [shard 0] osd - PG 2.0 stop 2022-03-29T17:57:17.600 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,587 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.600 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,587 [shard 0] osd - PG 2.1 stop 2022-03-29T17:57:17.600 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,587 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.600 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 2.3 stop 2022-03-29T17:57:17.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.0 stop 2022-03-29T17:57:17.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.601 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.1 stop 2022-03-29T17:57:17.602 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.602 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.2 stop 2022-03-29T17:57:17.602 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.602 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.3 stop 2022-03-29T17:57:17.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.4 stop 2022-03-29T17:57:17.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.5 stop 2022-03-29T17:57:17.603 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.7 stop 2022-03-29T17:57:17.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.8 stop 2022-03-29T17:57:17.604 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.605 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.9 stop 2022-03-29T17:57:17.605 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.605 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.e stop 2022-03-29T17:57:17.605 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.606 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.11 stop 2022-03-29T17:57:17.606 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.606 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.14 stop 2022-03-29T17:57:17.606 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.607 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.16 stop 2022-03-29T17:57:17.607 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.607 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.18 stop 2022-03-29T17:57:17.607 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.19 stop 2022-03-29T17:57:17.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.1a stop 2022-03-29T17:57:17.608 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.1d stop 2022-03-29T17:57:17.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - PG 3.1e stop 2022-03-29T17:57:17.609 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - osdmap::stop 2022-03-29T17:57:17.610 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 2.0_head 2022-03-29T17:57:17.610 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.4_head 2022-03-29T17:57:17.610 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.7_head 2022-03-29T17:57:17.610 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.9_head 2022-03-29T17:57:17.611 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.1d_head 2022-03-29T17:57:17.611 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.11_head 2022-03-29T17:57:17.611 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.5_head 2022-03-29T17:57:17.612 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 2.3_head 2022-03-29T17:57:17.612 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.19_head 2022-03-29T17:57:17.612 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.e_head 2022-03-29T17:57:17.612 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 2.1_head 2022-03-29T17:57:17.613 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.2_head 2022-03-29T17:57:17.613 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.1a_head 2022-03-29T17:57:17.613 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.14_head 2022-03-29T17:57:17.614 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.18_head 2022-03-29T17:57:17.614 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.16_head 2022-03-29T17:57:17.614 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.0_head 2022-03-29T17:57:17.614 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.1_head 2022-03-29T17:57:17.615 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.1e_head 2022-03-29T17:57:17.615 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.3_head 2022-03-29T17:57:17.615 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] osd - ReplicatedBackend::stop 3.8_head 2022-03-29T17:57:17.615 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] monc - stop 2022-03-29T17:57:17.616 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] monc - close 2022-03-29T17:57:17.616 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,588 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] closing: reset no, replace no 2022-03-29T17:57:17.616 INFO:tasks.ceph.osd.2.smithi138.stderr:DEBUG 2022-03-29 17:57:17,588 [shard 0] monc - reopen_session mon connection attempts complete 2022-03-29T17:57:17.616 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,589 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:17.617 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,589 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mon.? v2:172.21.15.138:3300/0] write_event: dropped 2022-03-29T17:57:17.617 INFO:tasks.ceph.osd.2.smithi138.stderr:WARN 2022-03-29 17:57:17,589 [shard 0] monc - cannot establish the active_con with any mon 2022-03-29T17:57:17.617 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,589 [shard 0] mgrc - stop 2022-03-29T17:57:17.617 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,589 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] closing: reset no, replace no 2022-03-29T17:57:17.618 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,589 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> mgr.? v2:172.21.15.138:6812/35195] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:17.618 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,589 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] closing: reset no, replace no 2022-03-29T17:57:17.618 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,589 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] closing: reset no, replace no 2022-03-29T17:57:17.619 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,589 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:17.619 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,590 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.0 v2:172.21.15.138:6804/35577] write_event: dropped 2022-03-29T17:57:17.619 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,590 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] write_event: dropped 2022-03-29T17:57:17.620 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,590 [shard 0] ms - [osd.2(cluster) v2:172.21.15.138:6802/35579 >> osd.1 v2:172.21.15.138:6801/35578] execute_wait(): protocol aborted at CLOSING -- std::system_error (error crimson::net:6, protocol aborted) 2022-03-29T17:57:17.620 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,590 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4179 172.21.15.138:0/2702680834@41200] closing: reset no, replace no 2022-03-29T17:57:17.620 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,590 [shard 0] ms - [osd.2(client) v2:172.21.15.138:6803/35579 >> client.4179 172.21.15.138:0/2702680834@41200] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2022-03-29T17:57:17.620 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,591 [shard 0] osd - Heartbeat::Peer: osd.1 removed 2022-03-29T17:57:17.621 INFO:tasks.ceph.osd.2.smithi138.stderr:INFO 2022-03-29 17:57:17,595 [shard 0] osd - crimson shutdown complete 2022-03-29T20:25:35.836 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-29T20:25:35.874 DEBUG:teuthology.task.console_log:Killing console logger for smithi138 2022-03-29T20:25:35.875 DEBUG:teuthology.exit:Finished running handlers